1
Lecture XI
QuasiNewton Methods of Optimization
I.
A Baseline Scenario
A.
In this lecture, we develop several alternatives to the NewtonRaphson
algorithm.
As starting point, I want to discuss a prototype algorithm.
Algorithm U (Model algorithm for ndimensional unconstrained
minimization).
Let
x
k
be the current estimate of
x
*
(the point
which
minimizes the objective function
f
(
x
)).
U1. [Test for convergence] If the conditions for convergence are satisfied,
the algorithm terminates with
x
k
as the solution.
U2. [Compute a search direction] Compute a nonzero
n
vector
p
k
, the
direction of the search.
U3. [Compute a step length] Compute a scalar a
k
, the step length, for
which
f
(
x
k
+
a
k
p
k
)<
f
(
x
k
).
U4. [Update the estimate of the minimum] Set
x
k
+1
=
x
k
+
a
k
p
k
,
k
=
k
+1,
and go back to step U1.
B.
Given the steps to the prototype algorithm, I want to develop a sample
problem that we can compare the various algorithms against.
Notebook
Algorithm3.ma includes the numeric problem:
max
.
.
.
.
x
x
x x
x
st x
x
x
x
1
2
2
3
3
4
4
1
1
2
3
4
100
2
3
=
−
−
−
Using NewtonRaphson, the optimal point for this problem is found in 10
iterations using 1.23 seconds on the DEC Alpha.
II.
An Overview of Newton and QuasiNewton Algorithms
A.
The NewtonRaphson methodology can be used in U2 in the prototype
algorithm.
Specifically, the search direction can be determined by:
(
)
p
f
x
f
x
k
xx
k
x
k
= − ∇
∇
−
2
1
(
)
(
)
B.
QuasiNewton algorithms involve an approximation to the Hesian matrix.
For
example, we could replace the Hessian matrix with the negative of the identity
matrix for the maximization problem.
In this case the search direction would
be:
(
)
p
I n
f
x
k
x
k
= − −
∇
(
)
(
)
where
I
(
n
) is the identity matrix conformable with the gradient vector.
This
replacement is referred to as the steepest descent method.
In our sample
problem, this methodology requires 990 iterations and 29.28 seconds on the
DEC Alpha. This result highlights some interesting features regarding the
QuasiNewton methods:
1.
The steepest descent method requires more overall iterations.
In this
example, the steepest descent method requires 99 times as many
iterations as the NewtonRaphson method.
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
This is the end of the preview.
Sign up
to
access the rest of the document.
 Fall '08
 Moss
 Gradient descent, conjugate gradient, Newton's method in optimization, Bk sk

Click to edit the document details