ON TOPICS OPTIMIZATION
A Class Note For EE 840 Mathematical Methods in Engineering by Nurul Chowdhury Department of Electrical Engineering University of Saskatchewan
EE 840 Mathematical Methods in Engineering
1. Convex Sets
Convex sets are widely used in the formulation and solution of optimization problems. A convex set is a set of elements from a vector space such that all the points on the straight line between any two points of the set are also contained in the set. If a and b are points in a vector space the points on the straight line between a and b are given by
x = a + (1 - )b , 0 1
The above definition can be restated as: A set S is convex if for any two points a and b belonging to S there are no points on the line between a and b that are not members of S. Another restatement of the definition is: A set S is convex if there are no points a and b in S such that there is a point on the line between a and b that does not belong to S. The expression a + (1 - )b , 0 1 is called a convex combination of the points a and b. In general a convex combination of x1 , x 2 , x 3 , L , x n is a point x of the form x = 1 x1 + 2 x 2 + L + n x n where 1 + 2 + L + n = 1 and i 0, i = 1, 2, L , n Some important theorems on convex sets are listed below. Theorem: A hyperplane is a convex set. Theorem: The intersection of a finite number of convex sets is also a convex set. Theorem: The set C of all convex combination of points x1 , x 2 , x 3 , L , x n is a convex set.
2. Mean Value Theorem
Mean Value Theorem: Let f be a function which is differentiable on the closed interval [a, b]. Then there exists a point c in (a, b) such that
f (b ) - f (a ) b-a Mean value theorem indicates that the slope of f ( x ) at c as shown in Figure 1will be the same as the slope of the line segment connecting the two points [a , f (a )] and [b , f (b )] . The implication of mean value theorem is significant if we want to evaluate f (b ) based on the given value of f ( x ) at a. It implies that if the point c is found then no other derivative of f ( x ) higher than f ( x ) would be required to evaluate f (b ) . The concept can be extended to higher order derivatives if we assume that f ( x ) itself represents the nth derivative of a function. c can be expressed as a convex combination of a and b in the following manner. c = a + (1 - )b 0 1 f (c ) =
N. Chowdhury Department of Electrical Engineering
Page 2 of 27
EE 840 Mathematical Methods in Engineering
y
b a c x
Figure 1: Display of Mean Value Theorem
3. Optimization of Functions of a Single Variable
If f ( x ) has a relative maximum at a point x0 then we know from the definition of relative maximum that in the neighbourhood , about x0 , f ( x ) f ( x0 ) . If h is small, then we can write
f ( x0 + h ) - f ( x0 ) 0 ......... (1)
Expression (1) can be divided by h and therefore,
f (x 0 + h ) - f ( x0 ) 0 h f (x 0 + h ) - f ( x0 ) 0 h h > 0 ..........(2) h < 0 ...........(3)
df ( x0 ) 0 and similarly dx
If we take the limit as h 0 , then we obtain from Eqn. (2) that from Eqn. (3) that
df ( x0 ) 0 . These two results together lead to the conclusion that dx
df ( x0 ) = 0 . An analogous argument (with inequalities reversed) can be made for a dx df ( x0 ) minimum point and it too yields = 0. dx
N. Chowdhury Department of Electrical Engineering
Page 3 of 27
EE 840 Mathematical Methods in Engineering
A simple explanation can be given for the necessary condition for a maximum as show in Figure 1. A function f ( x ) is shown in Figure 1. The slope, f ( x ) of this function is shown at several places on the curve with arrow marked line segments. f ( x ) is positive on the left-hand side of the maximum point and negative on the right-hand side of the maximum point. Since f ( x ) is continuous, a transition from positive to negative can only happen if f ( x ) becomes zero in between. As shown in Figure 2, f ( x ) is zero at x0 indicating that the function f ( x ) does not increase beyond x0 . A similar explanation with the signs reversed can be given for a minimum point.
Figure 2: A function with a maximum
3.1 Sufficient conditions for an Optimum point
Suppose that f ( x ) and its first two derivatives are continuous at x0 . Using Taylor's theorem we can write h2 (x 0 ) + f ( x0 + h ) = f ( x0 ) + hf f [x0 + (1 - )( x0 + h )] 0 1 ..... (4) 2 If f ( x ) has a relative maximum at x0 , then we know from the necessary condition that f ( x ) = 0 . Therefore Eqn. (4) can rewritten as:
h2 f (x0 + h ) - f (x0 ) = f [x0 + (1 - )( x0 + h )] 0 1 ...... (5) 2 If f ( x0 ) is a maximum point, then it follows that f (x0 + h ) - f (x0 ) = h2 f [x0 + (1 - )( x0 + h )] < 0 0 1 ..... (6) 2
h2 That means f [x 0 + (1 - )( x0 + h )] < 0 . Since h 2 is always positive, 2 [x0 + (1 - )( x0 + h )] < 0 f
N. Chowdhury Department of Electrical Engineering
Page 4 of 27
EE 840 Mathematical Methods in Engineering
It follows from the continuity of f ( x ) that if f [x0 + (1 - )( x0 + h )] < 0 , then f ( x0 ) < 0 . Therefore we conclude that if f ( x0 ) < 0 when f ( x0 ) = 0 , f ( x0 ) is a maximum. By an analogous argument that if f ( x0 ) > 0 when f ( x0 ) = 0 , f ( x0 ) is a minimum. A simple explanation for the second derivative condition can be given with respect to Figure 1. f ( x ) is positive on the left-hand side of the maximum point and negative on the right-hand side of the maximum point. This means that for a maximum, f ( x ) decreases as we move from the left side of the maximum point to the right side of the maximum point. Decrease of f ( x ) means its derivative with respect to x is negative, i.e., f ( x ) < 0 in the neighbourhood of x0 including at x0 . However, it is possible that at x0 , both the first derivative and the second derivative vanish. We must then examine the higher order derivatives. Let us start with the following theorem. Theorem: Assume that f ( x ) and its first n derivatives are continuous. Then f ( x ) has a relative maximum or minimum at x0 if and only if n is even, where n is the order of the first non-vanishing derivative at x0 . The function has a maximum at x0 if f and a minimum if f
n n
(x0 ) < 0
(x0 ) > 0 .
Proof: Using Taylor's expansion we can write
f ( x0 + h ) = f ( x0 ) + hf ( x 0 ) +
h2 h3 f ( x0 ) + f ( x0 ) + L 2! 3! h n -1 hn n L+ f n -1 ( x0 ) + f [x0 + (1 - )( x0 + h )] (n - 1)! n! 01
The first n - 1 derivatives of f ( x ) at x0 vanish, i.e.,
f ( x0 ) = f ( x0 ) = f ( x0 ) = LL = f
n -1
(x0 ) = 0 .
Therefore,
f (x0 + h ) = f (x0 ) +
or,
hn n f [x0 + (1 - )( x0 + h )] n!
hn n f [x0 + (1 - )( x0 + h )] n! The continuity of f n ( x ) assures us that f n [x0 + (1 - )( x0 + h )] will have same sign as f (x0 + h ) - f (x0 ) =
(x0 ) . This leads to the conclusion that f (x0 + h ) - f (x0 ) will have the same sign as f n ( x0 ) if n is even. Because of the continuity of f n ( x ) , f n [x0 + (1 - )( x0 + h )] have
f
n
N. Chowdhury Department of Electrical Engineering
Page 5 of 27
EE 840 Mathematical Methods in Engineering
the same sign of f n ( x0 ) regardless of h is positive or negative. Therefore, n must be even because that is the only situation where the sign of f ( x0 + h ) - f ( x0 ) is same as
hn n f [x0 + (1 - )( x 0 + h )] . And, thus f ( x0 + h ) - f ( x0 ) is positive when f n ( x0 ) is n! positive and f ( x0 + h ) - f ( x0 ) is negative when f n ( x0 ) is negative. f ( x0 ) will be a
minimum if f
n
(x0 ) is positive and will be a maximum if
4
f
n
(x0 ) is negative.
Let us consider the following function.
z = f ( x ) = x 4 - 4 x 3 + 6 x 2 - 4 x + 1 = ( x - 1)
dz d 2z d 3z d 4z 3 2 = 12( x - 1) ; = 4( x - 1) ; = 24( x - 1) ; = 24 dx dx 2 dx 3 dx 4
dz d 2 z d 3 z = = =0 dx dx 2 dx 3 d 4z d 4z The first non-vanishing derivative is . n is even and is positive, therefore, dx 4 dx 4 f ( x ) has a minimum at x = 1 . Figure 3 shows the plot of this function.
At x = 1 ,
Figure 3: Plot of f(x) = (x-1)4 Consider another function.
z = f ( x ) = x 3 - 3 x 2 + 3 x - 1 = ( x - 1)
3
dz d 2z d 3z 2 = 3( x - 1) ; = 6( x - 1) ; =6 dx dx 2 dx 3
N. Chowdhury Department of Electrical Engineering
Page 6 of 27
EE 840 Mathematical Methods in Engineering
The first non-vanishing derivative is the third derivative. That means n is odd and, therefore, we have neither a maximum nor a minimum. This should be obvious if we plot 3 z. Figure 4 shows the plot of z = f ( x ) = x 3 - 3 x 2 + 3 x - 1 = ( x - 1) . Notice a point of inflection at x = 1 .
Figure 4: Plot of f(x) = (x-1)3
3.2 Global Maximum (Minimum) of One Variable
The steps for finding the global maximum of a continuous function f ( x ) in the interval a x b are shown below. 1. Compute f (a ) and f (b ) . 2. Find an expression for f ( x ) . 3. Find the roots of f ( x ) = 0 . Assume that the roots are x1 , x 2 , x 3 , L , x n . 4. If the roots are not within [a ,b] , then the maximum is the larger of f (a ) and f (b ) . If the roots are within [a ,b] , then the maximum is the largest of f (a ) , f (b ) , f ( x1 ) , f ( x 2 ) , f ( x 3 ) , .........., f ( x n ) . Let us consider the following function.
f ( x ) = x 3 - x 2 - x + 3 for 0 x 2 f (0) = 3 and f (2) = 5 f ( x ) = 3 x 2 - 2 x - 1
N. Chowdhury Department of Electrical Engineering
Page 7 of 27
EE 840 Mathematical Methods in Engineering
Solving f ( x ) = 3 x 2 - 2 x - 1 = 0 we get x1 = - 1 and x 2 = 1 3
x1 is not in [0,2] , so we ignore it. But x 2 is in [0,2] , so we calculate f ( x 2 ) = f (1) = 2 . Therefore the maximum is Max[ f (0) = 3, f (2) = 5, f (1) = 2] = 5
Optima of Convex and Concave Functions
A convex function f ( x ) over a closed interval a x b satisfies the following condition:
f [x1 + (1 - )x 2 ] f ( x1 ) + (1 - ) f ( x 2 )
for any x1 , x 2 in [a ,b] and all , 0 1 . This means that any point on the line segment connecting ( x1 , f ( x1 )) and ( x 2 , f ( x 2 )) will be on or above the curve, f ( x ) . Figure 5 shows a convex function. A convex function can also be defined in another manner. If the second derivative of a function of one variable is positive within a range of values of the variable, then the function is convex in that range.
Figure 5: A convex function Theorem: Let f ( x ) be a convex function over a closed interval, a x b . Then any relative minimum of f ( x ) in this interval is also the absolute or global minimum of f ( x ) over the interval. The global maximum of a convex function f ( x ) over a closed interval, a x b will be taken on at either x = a or x = b or both.
N. Chowdhury Department of Electrical Engineering
Page 8 of 27
EE 840 Mathematical Methods in Engineering
A concave function f ( x ) over a closed interval a x b satisfies the following condition:
f [x1 + (1 - )x 2 ] f ( x1 ) + (1 - ) f ( x 2 )
for any x1 , x 2 in [a ,b] and all , 0 1 . This means that any point on the line segment connecting ( x1 , f ( x1 )) and ( x 2 , f ( x 2 )) will be on or below the curve, f ( x ) . Figure 6 shows a concave function. A concave function can also be defined in another manner. If the second derivative of a function of one variable is negative within a range of values of the variable, then the function is concave in that range.
Figure 6: A concave function Theorem: Let f ( x ) be a concave function over a closed interval, a x b . Then any relative maximum of f ( x ) in this interval is also the absolute or global maximum of f ( x ) over the interval. The global minimum of a concave function f ( x ) over a closed interval, a x b will be taken on at either x = a or x = b or both.
4. Optimization of Functions of Several Variables
In an n-dimensional case, x = ( x1 , x 2 , x 3 , L , x n ) indicates a point or vector in an Euclidean space R n . f ( x ) will be used to designate f ( x1 , x 2 , x 3 , L , x n ) . A function
N. Chowdhury Department of Electrical Engineering
Page 9 of 27
EE 840 Mathematical Methods in Engineering
f ( x ) is convex over some convex set X in R n if for any two points x1 and x 2 in X and for all , 0 1 f [x1 + (1 - )x 2 ] f ( x1 ) + (1 - ) f ( x 2 )
In similar manner a function f ( x ) is concave over some convex set X in R n if for any two points x1 and x 2 in X and for all , 0 1
f [x1 + (1 - )x 2 ] f ( x1 ) + (1 - ) f ( x 2 )
It should be mentioned at this point that the sum of convex functions is a convex function and the sum of concave functions is a concave function. Using Taylor's theorem, if f ( x ) is continuous and has continuous first partial derivatives over an open convex set X in R n , then for any two points x1 and x 2 = x1 + h in X, there exists a , 0 1 such that
f ( x 2 ) = f ( x1 ) + f (x1 + (1 - )x 2 )h
provided h is small. f is the gradient vector and is defined by
f f f f = x , x , L x 2 n 1
We can extend Taylor's theorem in the n-dimensional case as we did in the onedimensional case. If f ( x ) is continuous and has continuous first- and second-order partial derivatives over an open convex set X in R n , then for any two points x1 and x 2 = x1 + h in X, there exists a , 0 1 such that
f ( x 2 ) = f ( x1 ) + f ( x1 )h + 1 h T H [x1 + (1 - )x 2 ]h 2
H is the Hessian matrix of f ( x ) and is defined as a matrix of n 2 second partial derivatives of f ( x ) .
2 f 2 x1 2 f H = x x 2 1 L 2 f x n x1
2 f x1 x 2 2 f 2 x 2 L 2 f x n x 2
2 f x1 x n 2 f L x 2 x n L L 2 f L 2 x n L
For a multivariable case, for a minimum or maximum at x = x0
f ( x0 ) = 0 i = 1, 2 , 3, L , n xi
N. Chowdhury Department of Electrical Engineering
Page 10 of 27
EE 840 Mathematical Methods in Engineering
f ( x0 + h ) = f ( x0 ) + f ( x0 )h + 1 h T H [x0 + (1 - )( x0 + h )]h 2
or, f ( x0 + h ) - f ( x0 ) = 1 h T H [x0 + (1 - )( x0 + h )]h 2 Since we have assumed the existence and continuity of the second partial derivatives, 2 f (x 0 ) 2 f will have the same sign as [x0 + (1 - )(x0 + h )] . Therefore, xi x j xi x j
f ( x0 + h ) - f ( x0 ) will be +ive if h T H [x0 ]h is +ive and ive if h T H [x0 ]h is ive. f ( x )
will be a minimum at x0 if h T H [x0 ]h is positive. f ( x ) will be a maximum at x0 if
h T H [x0 ]h is negative.
However, a quadratic form h T H [x0 ]h will be positive if and only if the Hessian is a positive definite matrix. Similarly h T H [x0 ]h will be negative if and only if the Hessian is a negative definite matrix.
Example: Determine the maximum of
2 2 2 f ( x ) = f ( x1 , x 2 , x 3 ) = 16 x1 + 24 x 2 - 4 x1 - 3 x 2 - x 3
[f (x )]
T
f ( x ) x1 - 8 x1 + 16 0 f ( x ) = = - 6 x 2 + 24 = 0 x 2 f ( x ) - 2 x 3 0 x 3
2 Therefore, x0 = 4 . The Hessian matrix H ( x ) can be determined as: 0 2 f (x ) = -8 x12 2 f (x ) = -6 2 x 2 2 f (x ) =0 x1 x 2 2 f (x ) =0 x 2 x 3 2 f (x ) =0 x1 x 3 2 f (x ) = -2 2 x 3
0 0 - 8 0 -6 H (x0 ) = 0 0 0 - 2
We have to prove whether H ( x0 ) is positive definite or negative definite.
N. Chowdhury Department of Electrical Engineering
Page 11 of 27
EE 840 Mathematical Methods in Engineering
y H ( x 0 ) y = [ y1
T
y2
0 0 y1 - 8 0 -6 y 3 ] 0 y 2 0 0 - 2 y 3
2 2 2 y T H ( x0 ) y = -8 y1 - 6 y 2 - 2 y 3 which is clearly less than zero for any
y T = [ y1
y2
y 3 ] provided y 0 . Therefore, x0 = [2 4 0] is a maximizing point.
T
Example: Determine the minimum of
2 2 2 f ( x ) = f ( x1 , x 2 , x 3 ) = 2 x1 + 3 x 2 + 4 x 3 - 8 x1 - 12 x 2 - 24 x 3 + 110
[f (x )]
T
f ( x ) x1 4 x1 - 8 0 f ( x ) = 6 x - 12 = 0 = x 2 2 f ( x ) 8 x 3 - 24 0 x 3
2 Therefore, x0 = 2 . The Hessian matrix H ( x ) can be determined as: 3
2 f (x ) =4 2 x1 2 f (x ) =6 2 x 2
2 f (x ) =0 x1 x 2 2 f (x ) =0 x 2 x 3
2 f (x ) =0 x1 x 3 2 f (x ) =8 2 x 3
4 0 0 H ( x 0 ) = 0 6 0 0 0 8
We have to prove whether H ( x0 ) is positive definite or negative definite.
y H ( x 0 ) y = [ y1
T
y2
4 0 0 y 1 y 3 ]0 6 0 y 2 0 0 8 y 3
2 2 2 y T H ( x0 ) y = 4 y1 + 6 y 2 + 8 y 3 which is clearly greater than zero for any
y T = [ y1
y2
y 3 ] provided y 0 . Therefore, x0 = [2 2 3] is a minimum point.
T
N. Chowdhury Department of Electrical Engineering
Page 12 of 27
EE 840 Mathematical Methods in Engineering
4.1 Constrained Optimization Equality Constraint
An optimization problem with equality constraints (often referred to as hard constraints) can be specified as
max imize z = f ( x ) = f ( x1 , x 2 , L , x n ) ....... (7) subject to g i ( x ) = g i ( x1 , x 2 , L , x n ) = bi i = 1, 2 , L , m
......(8)
The Lagrangian function corresponding to Eqns. (7) and (8) is
F (x , ) = f ( x ) + i [bi - g i ( x )] ......... (9)
i =1 m
The necessary conditions for a point x * , *
[
]
T
to maximize F (x , ) are,
g ( x ) F (x , ) f ( x ) m = - i i =0 x k x k x k i =1 F (x , ) = bi - g i ( x ) = 0 i Example:
2 2 min imize z = 3 x1 + 4 x 2
k = 1, 2, L , n ......... (10)
i = 1, 2, L , m ........... (11)
subject to 2 x1 - 3 x 2 = 10
First, form the Lagrangian function as:
2 2 F ( x1 , x 2 , ) = 3 x1 + 4 x 2 + (10 - 2 x1 + 3 x 2 )
F ( x , ) = 6 x1 - 2 = 0 .................. (12.a) x1 F ( x , ) = 8 x 2 + 3 = 0 ...................(12.b) x 2 F ( x , ) = 10 - 2 x1 + 3 x 2 = 0 ................(12.c)
Solving Eqns. (12.a), (12.b) and (12.c) we get
x1 = 80 43 x2 = - 90 43 = 240 43
Therefore, the corresponding minimum value of z is: 27.907
N. Chowdhury Department of Electrical Engineering
Page 13 of 27
EE 840 Mathematical Methods in Engineering
Inequality Constraint
Many optimization problems come with inequality constraints (often referred to as soft constraints). The general approach is to convert all inequality constraints to equality constraints and form a Lagrangian function and follow the same technique as discussed earlier. Assume we have the following problem:
max imize
f ( x1 , x 2 )
subject to g ( x1 , x 2 ) 0
Convert the inequality constraint to an equality constraint by introducing a new variable, s (often referred to as slack variable) in the following manner:
g ( x1 , x 2 ) - s 2 = 0
The variable s is squared to ensure that s 2 is always positive regardless the value of s. With this modification, the equivalent problem becomes:
min imize
f ( x1 , x 2 )
subject to g ( x1 , x 2 ) - s 2 = 0
The Lagrangian function is:
F ( x1 , x 2 , , s ) = f ( x1 , x 2 ) - g ( x1 , x 2 ) - s 2
F f g = - = 0 .......... (13.a) x1 x1 x1 F f g = - = 0 ......... (13.b) x 2 x 2 x 2 F = - g ( x1 , x 2 ) + s 2 = 0 .......... (13.c) F = 2s = 0 ................................ (13.d) s
[
]
We take partial derivatives of the Lagrangian function and equate them to zero.
Condition (13.d) states that if 2s = 0 , either or s or both equal to zero. If s is zero then from condition (13.c) we can write that g ( x1 , x 2 ) = 0 . Multiply both sides of condition (13.d) by s. We get
s 2 = 0
i.e., g ( x1 , x 2 ) = 0 ............. (13.e) The essential conditions, then are (13.a), (13.b), (13.e) and the original inequality constraint. These conditions are known as Kuhn-Tucker conditions for the optimization of a function subject to an inequality constraint. N. Chowdhury Page 14 of 27
Department Electrical of Engineering
EE 840 Mathematical Methods in Engineering
The Kuhn-Tucker conditions are summarized in the following: (a) (b) (c) (d)
f g - =0 x1 x1 f g - =0 x 2 x 2
g ( x1 , x 2 ) = 0
g ( x1 , x 2 ) 0
Example:
min imize f ( x1 , x 2 ) = 2 x1 + x1 x 2 + 3 x 2
2 subject to x1 + x 2 3
The inequality constraint can be rewritten as:
2 g ( x1 , x 2 ) = x1 + x 2 - 3 0
Referring to the Kuhn-Tucker conditions we can write (a) (b) (c)
2 + x 2 - 2x1 = 0 3 + x1 - = 0
2 (x1 + x 2 - 3) = 0
First, we will assume that the constraint will be satisfied. In that case we will then set = 0 and solve (a) and (b) for x1 and x 2 . We get x1 = -3 x 2 = -2 . We check to see if the constraint is satisfied. In this case the constraint is satisfied and, therefore, x1 = -3 x 2 = -2 will be our minimum point. The minimum of the function will be -6. If after solving (a) and (b) for x1 and x 2 we find that the constraint is violated then we will set the inequality constraint as an equality constraint and set 0 . In that case we will have to solve (a), (b) and (c) for x1 , x 2 and .
4.2 Multivariate Grid Search Method
In this method the region is divided into a grid structure. The function is evaluated at each node of this grid structure. The movement from one node to another is made in the direction of increase (decrease) of the function. This is relatively an inefficient but straightforward method. The method is useful as long as the dimensionality is not too great. This method can be implemented in the following manner.
N. Chowdhury Department of Electrical Engineering
Page 15 of 27
EE 840 Mathematical Methods in Engineering
1.
Place a grid of some pre-selected size, xi , i = 1, 2, L , n in each variable xi in the n-dimensional region, ai xi bi , i = 1, 2, L , n over which we want to optimize f ( x ) .
2. 3. 4. 5. 6.
Select (by some means) a node of this grid as the starting point. Evaluate the function, f ( x ) at the 3 n - 1 surrounding points, where n is the dimensionality of x . Select the point with the greatest (least) functional value, f ( x ) . This point, ^ , becomes the new starting point. x Repeat steps 3 and 4 until the central point yields the greatest (least) value of f (x ) . Reduce the grid size by halving or some other process and return to step 3. This process of grid size reduction is continued until some pre-specified tolerance is reached.
Figure 7 shows the grid structure in a two dimensional case. The initial point is labeled as A. The 3 2 - 1 points surrounding A are numbered 1 to 8. If the contours are in the direction of the function increasing towards B, then if we are maximizing, we would choose point 3 as our next starting point. Successive choices are shown as circled points. When we come close to the point B we would have to reduce the grid size in order to get as close as desired to the maximal point.
N. Chowdhury Department of Electrical Engineering
Page 16 of 27
EE 840 Mathematical Methods in Engineering
Figure 7: Two-dimensional grid search
4.3 Univariate Search Method
Due to the computational burden associated with changing all variables simultaneously, it is sometimes convenient to change one variable at a time. In an univariate search, one variable is changed such that the function is maximized in each of the coordinate T direction. It is assumed that x = [ y1 , y 2 , y 3 , L , y n ] . The steps associated with an univariate search can be described in the following manner: 1. 2. Start at some arbitrary point, x0 within the feasible solution space. Find the next point, x1 by performing a maximization with respect to the first variable, y1 , i.e.,
x1 = x0 + 1 1
where 1 = [1, 0 , 0 , L , 0] . 1 is a scalar such that f ( x0 + 1 1 ) is maximized.
T
3.
The general step corresponding to the k th variable, y k is: Page 17 of 27
N. Chowdhury Department of Electrical Engineering
EE 840 Mathematical Methods in Engineering
Find the next point, x k by performing a maximization with respect to the first variable, y k , i.e.,
x k = x k -1 + k k
such that f ( x k -1 + k k ) is maximized. 4. 5. Find the nth point by maximizing the function with respect to the nth variable. Repeat steps 2, 3 and 4 until k for each variable is less than a tolerance.
Figure 8 shows a univariate search approach for a two-dimensional case. The univariate approach works well when there is little interaction among the variables. However, the method could be ineffective if there is significant interaction among the variables.
x3
x1 x2
x0
Figure 8: A univariate search in a two-dimensional case Example:
2 2 Minimize Z = 3 x1 + 4 x 2 - 5 x1 x 2 - 2 x1
Let us start at ( x1 , x 2 ) = (5, 5) . Fix x 2 = 5 and solve the problem Min Z (x1 , 5) . We find that Min Z (x1 , 5) occurs at x1 = 4.5 . Next, we minimize Z (4.5, x 2 ) with respect to x 2 . The minimum occurs at x 2 = 2.8125 . At each stage we only solve a one-dimensional optimization problem. The results obtained by following this sequence are shown in Table 1. The coordinates of the minimum point to 4 decimal places are (0.6956, 0.4348) .
N. Chowdhury Department of Electrical Engineering
Page 18 of 27
EE 840 Mathematical Methods in Engineering
Table 1: Results of a univariate search
Point Function Minimized Variable Found Best Current Point
x0 = (5, 5) x1 = (4.5, 5) x 2 = (4.5 , 2.8125) x 3 = (2.6771, 2.8125) x4 = (2.6771, 1.6732) x5 = (1.7277 , 1.6732) x6 = (1.7277 , 1.0798) x7 = (1.2332, 1.0798) x8 = (1.2332, 0.7708)
Z ( x1 , 5) Z (4.5, x 2 ) Z ( x1 , 2.8125) Z (2.6771, x 2 ) Z (x1 , 1.6732) Z (1.7277 , x 2 ) Z (x1 , 1.0798) Z (1.2332, x 2 ) Z (x1 , 0.7708)
x1 = 4.5 x 2 = 2.8125 x1 = 2.6771 x 2 = 1.6732 x1 = 1.7277 x 2 = 1.0798 x1 = 1.2332 x 2 = 0.7708 x1 = 0.9757
(4.5, 5) (4.5, 2.8125) (2.6771, 2.8125) (2.6771, 1.6732) (1.7277 , 1.6732) (1.7277 , 1.0798) (1.2332, 1.0798) (1.2332, 0.7708) (0.9757 , 0.7708)
4.4 Gradient Methods: Directional Derivatives
In a multidimensional case, it is important to know the direction along which the rate of change of a function is the maximum. This can be addressed with the help of directional derivatives. The directional derivative of f ( x ) at x0 in the direction u is
Du f ( x0 ) = lim it
0
f ( x0 + u ) - f ( x0 )
The derivative of f ( x ) with respect to a direction u can be expressed in terms of partial derivatives in the following manner:
Du f ( x0 ) = f ( x0 )u Du f (x 0 ) =
u = 1 ...................... (14) u =1
f ( x0 ) uj j =1 x j
n
In order to find the direction u such that the rate of change of f ( x ) at a point x0 is a maximum, we have to maximize
N. Chowdhury Department of Electrical Engineering
Page 19 of 27
EE 840 Mathematical Methods in Engineering
f ( x0 ) uj j =1 x j
n n
subject to g (u ) = u 2 = 1 . j
j =1
The Lagrangian function can be written as:
F (u , ) =
n f ( x0 ) u j + 1 - u 2 ...................... (15) j j =1 x j j =1 n
Differentiating (15) we get
F f ( x0 ) = - 2u j = 0 u j x j
j = 1, 2, 3, L , n ................ (16)
n F = 1 - u 2 = 0 .................................... (17) j j =1
From Eqn. (16) we can write
uj =
1 f ( x0 ) 2 x j
j = 1, 2, 3, L , n ........................(18)
Substituting Eqn. (18) into Eqn. (17) we have
1-
j =1
n
1 f ( x0 ) = 0 ............................ (19) 42 x j
2
2 =
2 1 f ( x0 ) ..................................... (20) 4
=
1 f ( x0 ) .................................... (21) 2
Substituting Eqn. (21) into Eqn. (18) we get
u=
f (x0 ) f (x 0 )
...................... (22)
The plus sign gives the rate of maximum increase of f ( x ) and the minus sign gives the rate of maximum decrease. It is however, obvious from Eqn. (22) that the gradient vector gives the direction of maximum increase of the function f ( x ) .
4.5 Direction of the Steepest Ascent
N. Chowdhury Department of Electrical Engineering
Page 20 of 27
EE 840 Mathematical Methods in Engineering
The maximum of a function can be compared to the peak of a mountain. The objective in mountain climbing is to take steps so that one gains altitude. To reach the peak in the fewest number of steps, each step has to be taken in the direction of steepest ascent. Let us explain this with the help of the following unconstrained example.
Maximize
2 f ( X 1 , X 2 ) = 7 X 1 + 4 X 2 + X 1 X 2 - X 12 - X 2
The gradient is
f X 7 + X 2 - 2 X 1 f (X 1 , X 2 ) = 1 = ...................... (23) f 4 + X 1 - 2 X 2 X 2
Let us start our ascent from a point P0 = (8, 2) . At P0 , f ( X 1 , X 2 ) = 12 . The gradient vector of f ( X 1 , X 2 ) at P0 is:
7 + 2 - 2(8 ) - 7 V0 = = or - 7 x1 + 8 x 2 4 + 8 - 2(2) 8
The unit gradient vector, v0 would be v0 = -0.66 x1 + 0.75 x 2 Vector v0 extends one unit distance from the point P0 (Figure 9) in the direction of steepest ascent. Therefore, we should move in the direction of v0 . The question, however, is how many number of units in the direction of v0 ? Because, with too many steps we may surpass the peak. Let us take an arbitrary number of units in the direction of v0 , say 5 units. This will place us at the point P1 as shown in Figure 8.
P1 = 8 x1 + 2 x 2 + 5(- 0.66 x1 + 0.75 x 2 ) = 4.7 x1 + 5.7 x 2 or , (4.7 , 5.7 )
At P1 , f ( X 1 , X 2 ) = 28 . It is obvious that by moving from P0 to P1 we have gained altitude, and therefore, we are closer to the peak. It is, however, apparent from Figure 8 that by taking 5 units along the direction of v0 we have passed the peak. The gradient vector of f ( X 1 , X 2 ) at P1 is:
7 + 5.7 - 2(4.7 ) 3.3 V1 = = or 3.3 x1 - 2.7 x 2 4 + 4.7 - 2(5.7 ) - 2.7
The unit gradient vector, v1 would be v1 = 0.77 x1 - 0.63 x 2 Since in our previous step we have passed the peak, we will take a shorter step this time along the direction of v1 , say one multiple of the unit vector. We will move from P1 to a new point P2 :
P2 = 4.7 x1 + 5.7 x 2 + (0.77 x1 - 0.63 x 2 ) = 5.5 x1 + 5.1x 2
N. Chowdhury Department of Electrical Engineering
Page 21 of 27
EE 840 Mathematical Methods in Engineering
At P2 , f ( X 1 , X 2 ) = 30.6 . The maximum of the function, f ( X 1 , X 2 ) = 31 occurs at (6, 5) . It is apparent that we have moved too far in the direction of the first gradient and not far enough in the direction of the second gradient. We need some technique to determine how far to move at each step without overshooting the peak.
2 f ( X 1 , X 2 ) = 7 X 1 + 4 X 2 + X 1 X 2 - X 12 - X 2
Figure 9: Gradient direction
4.6 Steepest Ascent Method: One Variable at a Time
Steepest ascent method utilizes gradient without overshooting the peak of a function. Let us use the previous example to illustrate the method.
Maximize
2 f ( X 1 , X 2 ) = 7 X 1 + 4 X 2 + X 1 X 2 - X 12 - X 2
The gradient of f ( X 1 , X 2 ) is:
f X 7 + X 2 - 2 X 1 f (X 1 , X 2 ) = 1 = f 4 + X 1 - 2 X 2 X 2
The gradient vector of f ( X 1 , X 2 ) at P0 = (8, 2) is:
N. Chowdhury Department of Electrical Engineering
Page 22 of 27
EE 840 Mathematical Methods in Engineering
f X - 7 V0 = 1 = f 8 X 2
The partial derivative of f ( X 1 , X 2 ) at P0 = (8, 2) indicates that the function, f ( X 1 , X 2 ) is decreasing at a rate of 7 units per unit increase in X 1 and increasing at a rate of 8 units per unit increase in X 2 . Therefore, in order to move away from P0 = (8, 2) it would be advantageous for us to decrease X 1 and increase X 2 . ( If we were to minimize the function, we would do the opposite.) If ...