This preview shows pages 1–3. Sign up to view the full content.
Convex Optimization Overview
Zico Kolter (updated by Honglak Lee)
October 17, 2008
1
Introduction
Many situations arise in machine learning where we would like to
optimize
the value of
some function. That is, given a function
f
:
R
n
→
R
, we want to Fnd
x
∈
R
n
that minimizes
(or maximizes)
f
(
x
). We have already seen several examples of optimization problems in
class: leastsquares, logistic regression, and support vector machines can all be framed as
optimization problems.
It turns out that, in the general case, Fnding the global optimum of a function can be a
very di±cult task. However, for a special class of optimization problems known as
convex
optimization problems
, we can e±ciently Fnd the global solution in many cases. Here,
“e±ciently” has both practical and theoretical connotations: it means that we can solve
many realworld problems in a reasonable amount of time, and it means that theoretically
we can solve problems in time that depends only
polynomially
on the problem size.
The goal of these section notes and the accompanying lecture is to give a very brief
overview of the Feld of convex optimization. Much of the material here (including some
of the Fgures) is heavily based on the book
Convex Optimization
[1] by Stephen Boyd and
Lieven Vandenberghe (available for free online), and EE364, a class taught here at Stanford
by Stephen Boyd. If you are interested in pursuing convex optimization further, these are
both excellent resources.
2
Convex Sets
We begin our look at convex optimization with the notion of a
convex set
.
Defnition 2.1
A set
C
is convex if, for any
x, y
∈
C
and
θ
∈
R
with
0
≤
θ
≤
1
,
θx
+ (1

θ
)
y
∈
C.
Intuitively, this means that if we take any two elements in
C
, and draw a line segment
between these two elements, then every point on that line segment also belongs to
C
. ²igure
1 shows an example of one convex and one nonconvex set. The point
+ (1

θ
)
y
is called
a
convex combination
of the points
x
and
y
.
1
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document Figure 1: Examples of a convex set (a) and a nonconvex set (b).
2.1
Examples
•
All of
R
n
.
It should be fairly obvious that given any
x, y
∈
R
n
,
θx
+ (1

θ
)
y
∈
R
n
.
•
The nonnegative orthant,
R
n
+
.
The nonnegative orthant consists of all vectors in
R
n
whose elements are all nonnegative:
R
n
+
=
{
x
:
x
i
≥
0
∀
i
=1
, . . . , n
}
. To show
that this is a convex set, simply note that given any
x, y
∈
R
n
+
and 0
≤
θ
≤
1,
(
+ (1

θ
)
y
)
i
=
i
+ (1

θ
)
y
i
≥
0
∀
i.
•
Norm balls.
Let
± · ±
be some norm on
R
n
(e.g., the Euclidean norm,
±
x
±
2
=
±
∑
n
i
=1
x
2
i
). Then the set
{
x
:
±
x
±≤
1
}
is a convex set. To see this, suppose
x, y
∈
R
n
,
with
±
x
1
,
±
y
1, and 0
≤
θ
≤
1. Then
±
+ (1

θ
)
y
±≤±
±
+
±
(1

θ
)
y
±
=
θ
±
x
±
+ (1

θ
)
±
y
1
where we used the triangle inequality and the positive homogeneity of norms.
This is the end of the preview. Sign up
to
access the rest of the document.