cs229-cvxopt - Convex Optimization Overview Zico Kolter...

Info iconThis preview shows pages 1–3. Sign up to view the full content.

View Full Document Right Arrow Icon
Convex Optimization Overview Zico Kolter (updated by Honglak Lee) October 17, 2008 1 Introduction Many situations arise in machine learning where we would like to optimize the value of some function. That is, given a function f : R n R , we want to Fnd x R n that minimizes (or maximizes) f ( x ). We have already seen several examples of optimization problems in class: least-squares, logistic regression, and support vector machines can all be framed as optimization problems. It turns out that, in the general case, Fnding the global optimum of a function can be a very di±cult task. However, for a special class of optimization problems known as convex optimization problems , we can e±ciently Fnd the global solution in many cases. Here, “e±ciently” has both practical and theoretical connotations: it means that we can solve many real-world problems in a reasonable amount of time, and it means that theoretically we can solve problems in time that depends only polynomially on the problem size. The goal of these section notes and the accompanying lecture is to give a very brief overview of the Feld of convex optimization. Much of the material here (including some of the Fgures) is heavily based on the book Convex Optimization [1] by Stephen Boyd and Lieven Vandenberghe (available for free online), and EE364, a class taught here at Stanford by Stephen Boyd. If you are interested in pursuing convex optimization further, these are both excellent resources. 2 Convex Sets We begin our look at convex optimization with the notion of a convex set . Defnition 2.1 A set C is convex if, for any x, y C and θ R with 0 θ 1 , θx + (1 - θ ) y C. Intuitively, this means that if we take any two elements in C , and draw a line segment between these two elements, then every point on that line segment also belongs to C . ²igure 1 shows an example of one convex and one non-convex set. The point + (1 - θ ) y is called a convex combination of the points x and y . 1
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Figure 1: Examples of a convex set (a) and a non-convex set (b). 2.1 Examples All of R n . It should be fairly obvious that given any x, y R n , θx + (1 - θ ) y R n . The non-negative orthant, R n + . The non-negative orthant consists of all vectors in R n whose elements are all non-negative: R n + = { x : x i 0 i =1 , . . . , n } . To show that this is a convex set, simply note that given any x, y R n + and 0 θ 1, ( + (1 - θ ) y ) i = i + (1 - θ ) y i 0 i. Norm balls. Let ± · ± be some norm on R n (e.g., the Euclidean norm, ± x ± 2 = ± n i =1 x 2 i ). Then the set { x : ± x ±≤ 1 } is a convex set. To see this, suppose x, y R n , with ± x 1 , ± y 1, and 0 θ 1. Then ± + (1 - θ ) y ±≤± ± + ± (1 - θ ) y ± = θ ± x ± + (1 - θ ) ± y 1 where we used the triangle inequality and the positive homogeneity of norms.
Background image of page 2
Image of page 3
This is the end of the preview. Sign up to access the rest of the document.

This document was uploaded on 10/19/2011.

Page1 / 14

cs229-cvxopt - Convex Optimization Overview Zico Kolter...

This preview shows document pages 1 - 3. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online