IE598-lecture13-case-study-on-logistic-regression.pdf - IE...

IE 598: Big Data Optimization Fall 2016
Lecture 13: Case Study on Logistic Regression – October 06
Lecturer: Niao He Scriber: Jialin Song
Overview: In this lecture we summarize algorithms we have covered so far for smooth convex optimization
problems. To better understanding the pros and cons of different algorithms, we conduct a case study on
logistic regression model to investigate the their empirical performances.
13.1 Summary
We summarize performance of algorithms on smooth convex optimization (including Interior Point Method
(IPM), Gradient Descent (GD), Accelerated Gradient Descent (AGD), Projection Gradient Descent (PGD),
Frank-Wolfe Algorithm (FW), Block Coordinated Gradient Descent (BCGD)) in terms of convergence rate,
iteration complexity and iteration cost in Table 13.1.
Table 13.1: Performance of algorithms on structured convex optimization and smooth convex optimization
Algorithm Convergence Rate Iteration Complexity Iteration Cost
Structured
Convex Optimization
(LP/SOCP/DCP)
IPM O ( ν exp( -
t
√
ν
)) O (
√
ν log(
1
)) one Newton step
Convex Strongly Convex Convex Strongly Convex
Smooth
Convex Optimization
GD
O (
LD
2
t
)
O ((
1 - κ
1+ κ
)
2 t
) O (
LD
2
)
L
μ
log(
1
) one gradient
AGD O (
LD
2
t
2
) O ((
1 -
√
κ
1+
√
κ
)
2 t
) O (
√
LD
√ ) O (
r
L
μ
log(
1
)) one gradient
PGD O (
LD
2
t
) O ((1 -
μ
L
)
t
) O (
LD
2
)
L
μ
log(
1
)
one gradient
one projection
FW O (
LD
2
t
) O (
LD
2
)
one gradient
one linear minimization
BCGD O (
bLD
2
)
t
O ((1 -
μ
bL
)
t
) O (
bLD
2
) O (
bL
μ
log (
1
))
O (1)-block gradient (randomized)
O ( b )-block gradient (cyclic)
O ( b )-block gradient (Gauss Southwell)
Notations:
• L is L-smooth constant
• μ is strong convexity parameter
• κ =
L
μ
ibs the condition number
• D is either k x
0
- x
*
k
2
or diameter of compact set X.
13.2 Logistic Regression
In this section, we first introduce the basic fundamentals of classification models and then formulate logistic
regression model.
13-1

Lecture 13: Case Study on Logistic Regression – October 06 13-2
13.2.1 Preliminaries on Classification Model
We consider a binary classification problem: Given ( x
1
, y
1
) , ( x
2
, y
2
) , ..., ( x
n
, y
n
), x
i
∈ R
d
, y
i
∈ {- 1 , 1 } . Note
that input x
i
is called feature vector, output y
i
is label. Suppose there is a family of function f ( x, w ), binary
classification aims to specify the best w such that x and y are related.
Based on the input feature vectors and certain type of function we adopt, the output can be predicted based
on certain prediction rules. For binary classification problem, the prediction rule can be shown as follows:
y =
1 , f ( x, w ) ≥ 0
- 1 , f ( x, w ) < 0
An error term can be defined as follows to denote whether the output is successfully predicted or not:
error =
1 , yf ( x, w ) < 0
0 , yf ( x, w ) ≥ 0
Obviously our goal is to find the best parameter w in order to minimize the total error in the process of
prediction. The optimization step in the binary classification model can be extracted below: