IE598-lecture16-smoothing-techniques-I.pdf - IE 598: Big...

IE 598: Big Data Optimization Fall 2016
Lecture 16: Smoothing Techniques I – October 18
Lecturer: Niao He Scribers: Harsh Gupta
Overview. We discussed Subgradient Descent and Mirror Descent algorithms for non-smooth convex op-
timization in the past week. We observed that Subgradient Descent is a special case of the Mirror Descent
algorithm. But, both these algorithms have general formulations and don’t exploit the structure of the prob-
lem at hand. In practice, we always know some thing about the structure of the optimization problem we
intend to solve. One can then utilize this structure to come up with more efficient algorithms as compared
to Subgradient Descent and Mirror Descent algorithms.
16.1 Introduction
We intend to solve the following optimization problem:
min
x ∈ X
f ( x ) (16.1)
where f is a convex but non-smooth, i.e., non-differentiable function, and X is a convex compact set. One
intuitive way to approach the above problem is to approximate the non-smooth function f ( x ) by a smooth
and convex function f
μ
( x ), so that we can use the standard techniques learnt so far in the course to solve
the problem. Hence, we want to reduce the problem in (16.1) to the following:
min
x ∈ X
f
μ
( x ) (16.2)
where f
μ
( x ) is a L
μ
-Lipschitz continuous, smooth and convex approximation of the function f ( x ). Now
we can use the techniques learnt earlier in this course like gradient descent, accelerated gradient descent,
Frank Wolfe algorithm, coordinate descent etc., to solve the above problem. Clearly, the objective now is to
come up with a reasonably good approximation f
μ
of f so that solving (16.2) is as close to solving (16.1) as
possible.
A motivation example: Consider the simplest non-smooth and convex function, f ( x ) = | x | . The following
function, known as the Huber function
f
μ
( x ) =
(
x
2
2 μ
, | x | ≤ μ
| x | -
μ
2
, | x | > μ
(16.3)
is a smooth approximation of the absolute value function. We plot the two functions (for μ = 1) in Figure
1. We make the following observations:
1. f
μ
( x ) is clearly continuous and differentiable everywhere. This can be seen straightforwardly from its
formulation given in (16.3).
2. We observe that f
μ
( x ) ≤ f ( x ). Also, f
μ
( x ) ≥ | x | -
μ
2
, therefore:
f ( x ) -
μ
2
≤ f
μ
( x ) ≤ f ( x )
Hence, if μ → 0, then f
μ
( x ) → f ( x ). Therefore, μ characterizes the approximation accuracy.
3. We also observe that | f
00
μ
( x ) | ≤
1
μ
. This implies that f
μ
( x ) is
1
μ
-Lipschitz continuous.
The Hubert function approximation has been widely used in machine learning to approximate non-smooth
loss functions, e.g. absolute loss (robust regression), hinge loss (SVM), etc.