Optimization Convex Boyd & Vandenberghe
1. Introduction
mathematical optimization least-squares and linear programming convex optimization example course goals and topics nonlinear optimization brief history of convex optimization
11
Mathematical optimization
(mathematical) optimization problem minimize f0(x) subject to fi(x) bi, x = (x1, . . . , xn): optimization variables f0 : Rn R: objective function fi : Rn R, i = 1, . . . , m: constraint functions
i = 1, . . . , m
optimal solution x has smallest value of f0 among all vectors that satisfy the constraints
Introduction
12
Examples
portfolio optimization variables: amounts invested in dierent assets constraints: budget, max./min. investment per asset, minimum return objective: overall risk or return variance device sizing in electronic circuits variables: device widths and lengths constraints: manufacturing limits, timing requirements, maximum area objective: power consumption data tting variables: model parameters constraints: prior information, parameter limits objective: measure of mist or prediction error
Introduction 13
Solving optimization problems
general optimization problem very dicult to solve methods involve some compromise, e.g., very long computation time, or not always nding the solution
exceptions: certain problem classes can be solved eciently and reliably least-squares problems linear programming problems convex optimization problems
Introduction
14
Least-squares
minimize solving least-squares problems analytical solution: x = (AT A)1AT b reliable and ecient algorithms and software computation time proportional to n2k (A Rkn); less if structured a mature technology using least-squares least-squares problems are easy to recognize a few standard techniques increase exibility (e.g., including weights, adding regularization terms)
Introduction 15
Ax b
2 2
Linear programming
minimize cT x subject to aT x bi, i solving linear programs no analytical formula for solution reliable and ecient algorithms and software computation time proportional to n2m if m n; less with structure a mature technology using linear programming not as easy to recognize as least-squares problems a few standard tricks used to convert problems into linear programs (e.g., problems involving 1- or -norms, piecewise-linear functions)
Introduction 16
i = 1, . . . , m
Convex optimization problem
minimize f0(x) subject to fi(x) bi,
i = 1, . . . , m
objective and constraint functions are convex: fi(x + y) fi(x) + fi(y) if + = 1, 0, 0 includes least-squares problems and linear programs as special cases
Introduction
17
solving convex optimization problems no analytical solution reliable and ecient algorithms computation time (roughly) proportional to max{n3, n2m, F }, where F is cost of evaluating fis and their rst and second derivatives almost a technology
using convex optimization often dicult to recognize many tricks for transforming problems into convex form surprisingly many problems can be solved via convex optimization
Introduction
18
Example
m lamps illuminating n (small, at) patches
lamp power pj
rkj kj
illumination Ik
intensity Ik at patch k depends on linearly lamp powers pj :
m
Ik =
j=1
akj pj ,
2 akj = rkj max{cos kj , 0}
problem: achieve desired illumination Ides with bounded lamp powers minimize maxk=1,...,n | log Ik log Ides| subject to 0 pj pmax, j = 1, . . . , m
Introduction 19
how to solve? 1. use uniform power: pj = p, vary p 2. use least-squares: minimize round pj if pj > pmax or pj < 0 3. use weighted least-squares: minimize
n k=1(Ik n k=1 (Ik
Ides)2
Ides)2 +
m j=1 wj (pj
pmax/2)2
iteratively adjust weights wj until 0 pj pmax 4. use linear programming: minimize maxk=1,...,n |Ik Ides| subject to 0 pj pmax, j = 1, . . . , m which can be solved via linear programming of course these are approximate (suboptimal) solutions
Introduction 110
5. use convex optimization: problem is equivalent to minimize f0(p) = maxk=1,...,n h(Ik /Ides) subject to 0 pj pmax, j = 1, . . . , m with h(u) = max{u, 1/u}
5
4
h(u)
3
2
1
0 0
1
u
2
3
4
f0 is convex because maximum of convex functions is convex exact solution obtained with eort modest factor least-squares eort
Introduction 111
additional constraints: does adding 1 or 2 below complicate the problem? 1. no more than half of total power is in any 10 lamps 2. no more than half of the lamps are on (pj > 0) answer: with (1), still easy to solve; with (2), extremely dicult moral: (untrained) intuition doesnt always work; without the proper background very easy problems can appear quite similar to very dicult problems
Introduction
112
Course goals and topics
goals 1. recognize/formulate problems (such as the illumination problem) as convex optimization problems 2. develop code for problems of moderate size (1000 lamps, 5000 patches) 3. characterize optimal solution (optimal power distribution), give limits of performance, etc. topics 1. convex sets, functions, optimization problems 2. examples and applications 3. algorithms
Introduction
113
Nonlinear optimization
traditional techniques for general nonconvex problems involve compromises local optimization methods (nonlinear programming) nd a point that minimizes f0 among feasible points near it fast, can handle large problems require initial guess provide no information about distance to (global) optimum global optimization methods nd the (global) solution worst-case complexity grows exponentially with problem size these algorithms are often based on solving convex subproblems
Introduction 114
Brief history of convex optimization
theory (convex analysis): ca19001970 algorithms 1947: simplex algorithm for linear programming (Dantzig) 1960s: early interior-point methods (Fiacco & McCormick, Dikin, . . . ) 1970s: ellipsoid method and other subgradient methods 1980s: polynomial-time interior-point methods for linear programming (Karmarkar 1984) late 1980snow: polynomial-time interior-point methods for nonlinear convex optimization (Nesterov & Nemirovski 1994)
applications before 1990: mostly in operations research; few in engineering since 1990: many new applications in engineering (control, signal processing, communications, circuit design, . . . ); new problem classes (semidenite and second-order cone programming, robust optimization)
Introduction 115