{[ promptMessage ]}

Bookmark it

{[ promptMessage ]}

This preview shows pages 1–6. Sign up to view the full content.

Quadratic Functions, Optimization, and Quadratic Forms Robert M. Freund February, 2004 1 2004 Massachusetts Institute of Technology.

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
2 1 Quadratic Optimization A quadratic optimization problem is an optimization problem of the form: T (QP) : minimize f ( x ) := 1 x T Qx + c x 2 n s.t. x . Problems of the form QP are natural models that arise in a variety of settings. For example, consider the problem of approximately solving an over-determined linear system Ax = b , where A has more rows than columns. We might want to solve: (P 1 ) : minimize Ax b n s.t. x . Now notice that Ax b 2 = x T A T Ax 2 b T Ax + b T b , and so this problem is equivalent to: (P 1 ) : minimize x T A T Ax 2 b T Ax + b T b n s.t. x , which is in the format of QP. A symmetric matrix is a square matrix Q n × n with the property that Q ij = Q ji for all i, j = 1 , . . . , n .
3 We can alternatively define a matrix Q to be symmetric if Q T = Q . We denote the identity matrix (i.e., a matrix with all 1’s on the diagonal and 0’s everywhere else) by I , that is, 1 0 . . . 0 0 1 . . . 0 I = . . . . . . . . . . . . , 0 0 . . . 1 and note that I is a symmetric matrix. The gradient vector of a smooth function f ( x ) : n is the vector of first partial derivatives of f ( x ): ∂f ( x ) ∂x 1 . f ( x ) := . . . ∂f ( x ) ∂x n The Hessian matrix of a smooth function f ( x ) : n is the ma- trix of second partial derivatives. Suppose that f ( x ) : n is twice differentiable, and let 2 f ( x ) [ H ( x )] ij := ∂x i ∂x j . Then the matrix H ( x ) is a symmetric matrix, reﬂecting the fact that 2 f ( x ) = 2 f ( x ) . ∂x i ∂x j ∂x j ∂x i A very general optimization problem is:

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
4 (GP) : minimize f ( x ) n s.t. x , where f ( x ) : n is a function. We often design algorithms for GP by building a local quadratic model of f ( · ) at a given point x = ¯ x . We form the gradient f x ) (the vector of partial derivatives) and the Hessian H x ) (the matrix of second partial derivatives), and approximate GP by the following problem which uses the Taylor expansion of f ( x ) at x = ¯ x up to the quadratic term. (P 2 ) : minimize f ˜ ( x ) := f x ) T ( x ¯ 2 ( x ¯ x )( x ¯ x ) + f x ) + 1 x ) T H x ) n s.t. x . This problem is also in the format of QP. Notice in the general model QP that we can always presume that Q is a symmetric matrix, because: 1 T x Qx = x T ( Q + Q T ) x 2 ¯ and so we could replace Q by the symmetric matrix Q := 1 2 ( Q + Q T ). Now suppose that f ( x ) := 1 x T Qx + c T x 2 where Q is symmetric. Then it is easy to see that: f ( x ) = Qx + c and H ( x ) = Q .
5 Before we try to solve QP, we first review some very basic properties of symmetric matrices. 2 Convexity, Definiteness of a Symmetric Matrix, and Optimality Conditions A function f ( x ) : n is a convex function if n f ( λx +(1 λ ) y ) λf ( x )+(1 λ ) f ( y ) for all x, y , for all λ [0 , 1] .

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}