Classification
Rong Statistical Jin
Classification Problems
Inpu t
Given
X
f : X Y
?
Y
Outpu t
input X={x1, x2, , xm}
Predict the class label y Y Y = {-1,1}, binary class classification problems Y = {1, 2, 3, , c}, multiple class classification problems Goal: need to learn the function: f: X Y
Examples of Classification Problem
Text categorization:
Doc: Months of campaigning and weeks of round-the-clock efforts in Iowa all came down to a final push Sunday,
Politics Topic: Non-politics
Input features X:
Word frequency {(campaigning, 1), (democrats, 2), (basketball, 0), } Y = +1: politics Y = -1: non-politics
Class label y:
Examples of Classification Problem
Text categorization:
Doc: Months of campaigning and weeks of round-the-clock efforts in Iowa all came down to a final push Sunday,
Politics Topic: Non-politics
Input features X:
Word frequency {(campaigning, 1), (democrats, 2), (basketball, 0), } Y = +1: politics Y = -1: not-politics
Class label y:
Examples of Classification Problem
Image Classification:
Which images are birds, which are not?
Input features X
Color histogram {(red, 1004), (red, 23000), } Y = +1: bird image Y = -1: non-bird image
Class label y
Examples of Classification Problem
Image Classification:
Which images are birds, which are not?
Input features X
Color histogram {(red, 1004), (blue, 23000), } Y = +1: bird image Y = -1: non-bird image
Class label y
Classification Problems
Inpu t
X
f : X Y
?
Y
Outpu t
Doc: Months of campaigning and weeks of round-the-clock efforts in Iowa all came down to a final push Sunday,
f: doc topic How to obtain f ? f: image topic
Politics Not-politics Birds Not-Birds
Learn classification function f from examples
Learning from Examples
Training examples: r r r Dtrain = { ( x1 , y1 ) , ( x2 , y2 ) ,..., ( xn , yn ) }
n : the number of training examples r xi d : ( xi ,1 , xi ,2 ,..., xi , d ) yi Y : Binary class: Y = {+1, -1} Multiple class: Y = {1, 2,..., c}
Identical Independent Distribution (i.i.d.)
Each training example is drawn independently from the identical source Training examples are similar to testing examples
Learning from Examples
Training examples: r r r Dtrain = { ( x1 , y1 ) , ( x2 , y2 ) ,..., ( xn , yn ) }
n : the number of training examples r xi d : ( xi ,1 , xi ,2 ,..., xi , d ) yi Y : Binary class: Y = {+1, -1} Multiple class: Y = {1, 2,..., c}
Identical Independent Distribution (i.i.d.)
Each training example is drawn independently from the identical source
Learning from Examples
Given training examples
r r r Dtrain = { ( x1 , y1 ) , ( x2 , y2 ) ,..., ( xn , yn ) }
Goal: learn a classification function f(x):XY that is consistent with training examples What is the easiest way to do it ?
K Nearest Neighbor (kNN) Approach
How many neighbors should we count ? (k=1) (k=4)
Cross Validation
Divide training examples into two sets
A training set (80%) and a validation set (20%)
Predict the class labels of the examples in the validation set by the examples in the training set Choose the number of neighbors k that maximizes the classification accuracy
Leave-One-Out Method
For k = 1, 2, , K Err(k) = 0; 1. Randomly select a training data point and hide its class label 2. Using the remaining data and given K to predict the class label for the left data point 3. Err(k) = Err(k) + 1 if the predicted label is different from the true label Repeat the procedure until all training examples are tested Choose the k whose Err(k) is minimal
Leave-One-Out Method
For k = 1, 2, , K Err(k) = 0; 1. Randomly select a training data point and hide its class label 2. Using the remaining data and given K to predict the class label for the left data point 3. Err(k) = Err(k) + 1 if the predicted label is different from the true label Repeat the procedure until all training examples are tested Choose the k whose Err(k) is minimal
Leave-One-Out Method
For k = 1, 2, , K Err(k) = 0; 1. Randomly select a training data point and hide its class label 2. Using the remaining data and given k to predict the class label for the left data point 3. Err(k) = Err(k) + 1 if the predicted label is different from the true label Repeat the procedure until all training examples are tested Choose the k whose Err(k) is minimal
(k=1)
Leave-One-Out Method
For k = 1, 2, , K Err(k) = 0; 1. Randomly select a training data point and hide its class label 2. Using the remaining data and given k to predict the class label for the left data point 3. Err(k) = Err(k) + 1 if the predicted label is different from the true label Repeat the procedure until all training examples are tested Choose the k whose Err(k) is minimal
(k=1)
Err(1) = 1
Leave-One-Out Method
For k = 1, 2, , K Err(k) = 0; 1. Randomly select a training data point and hide its class label 2. Using the remaining data and given k to predict the class label for the left data point 3. Err(k) = Err(k) + 1 if the predicted label is different from the true label Repeat the procedure until all training examples are tested Choose the k whose Err(k) is minimal
Err(1) = 1
Leave-One-Out Method
For k = 1, 2, , K Err(k) = 0; 1. Randomly select a training data point and hide its class label 2. Using the remaining data and given k to predict the class label for the left data point 3. Err(k) = Err(k) + 1 if the predicted label is different from the true label Err(1) = 3
k=2
Err(2) = 2 Err(3) = 6
Repeat the procedure until all training examples are tested Choose the k whose Err(k) minimal
Probabilistic is interpretation of KNN
Estimate the probability density function Pr(y|x) around the location of x
Count of data points in class y in the neighborhood of x A small neighborhood large variance unreliable estimation A large neighborhood large bias inaccurate estimation
Bias and variance tradeoff
Weighted kNN
Weight the contribution of each close neighbor based on their distances Weight function r r x xi 2 rr 2 w( x , xi ) = exp 2 2
Prediction
rr r i w( x , xi ) ( y, yi ) Pr( y | x ) = rr i w( x , xi )
1 ( y , yi ) = 0
y = yi y yi
Estimate 2 in the Weight Function
Leave one cross validation Training dataset D is divided into two sets
r Compute the Pr( y1 | x1 , D1 )
r Validation set ( x1 , y1 ) r r r Training set D1 = { ( x2 , y2 ), ( x3 , y3 ),..., ( xn , yn )}
Estimate 2 in the Weight Function
r Pr( y1 | x1 , D1 ) = r r w( x1 , xi ) ( y1 , yi )
i =2 n
r r w( x1 , xi )
i =2
n
r r x1 xi 2 2 exp 2 2 ( y1 , yi ) i=2 = r r n x1 xi 2 exp 2 2 2 i =2
n
Pr(y|x1, D-1) is a function of 2
Estimate 2 in the Weight Function
r Pr( y1 | x1 , D1 ) = r r w( x1 , xi ) ( y1 , yi )
i =2 n
r r w( x1 , xi )
i =2
n
r r x1 xi 2 2 exp 2 2 ( y1 , yi ) i=2 = r r n x1 xi 2 exp 2 2 2 i =2
n
Pr(y|x1, D-1) is a function of 2
Estimate 2 in the Weight Function
r In general, we can have expression for Pr( yi | xi , Di ) r Validation set ( xi , yi ) r r r r Training set D = { ( x , y ),...., ( x , y ), ( x , y )..., ( x , y )} i 1 2 i 1 i 1 i +1 i +1 n n
Estimate 2 by maximizing the likelihood
r l ( ) = log Pr( yi | xi , Di )
i =1 n
* = arg max l ( )
Estimate 2 in the Weight Function
r In general, we can have expression for Pr( yi | xi , Di ) r Validation set ( xi , yi ) r r r r Training set D = { ( x , y ),...., ( x , y ), ( x , y )..., ( x , y )} i 1 2 i 1 i 1 i +1 i +1 n n
Estimate 2 by maximizing the likelihood
r l ( ) = log Pr( yi | xi , Di )
i =1 n
* = arg max l ( )
Optimization
r r xi xk 2 exp 2 ( y k , yi ) n n 2 2 k i r l ( ) = log Pr( yi | xi , Di ) = log r r xi xk 2 i =1 i =1 i exp 2 2 2 k
= arg max l ( )
*
It is a DC (difference of two convex functions) function
Challenges in Optimization
Convex functions are easiest to be optimized Single-mode functions are the second easiest Multi-mode functions are difficult to be optimized
Gradient Ascent
r r xi xk 2 exp 2 ( y k , yi ) n n 2 2 k i r l ( ) = log Pr( yi | xi , Di ) = log r r xi xk 2 i =1 i =1 i exp 2 2 2 k
= 1/ 2
n n r l( ) = log Pr( yi | xi , Di ) = log i =1 i =1
(
k i k i
r r exp xi xk
i
) ( y , y ) r r exp ( x x )
2 2 k i 2 k 2
Gradient Ascent (contd)
n n r l( ) = log Pr( yi | xi , Di ) = log i =1 i =1
(
k i k i
r r exp xi xk
i
) ( y , y ) r r exp ( x x )
2 2 k i 2 k 2
Compute the derivative of l(), i.e., dl ( ) / d Update dl ( ) +t d How to decide the step size t?
Gradient Ascent: Line Search
Excerpt from the slides by Steven Boyd
Gradient Ascent
Stop criterion | dl ( ) / d |
is predefined small value
Start =0, Define , , and Compute dl ( ) / d
Choose step size t via backtracking line search Update + tdl ( ) / d Repeat ...