5525: CSCI Machine Learning (Spring 2012)
Supervised Learning
Rui Kuang
Department of Computer Science and Engineering
University of Minnesota
Noise and Model Complexity
Given similar training error,
use the simpler one
Simpler to use (lower
computational complexity)
Easier to train (lower space
complexity)
Easier to explain (more
interpretable)
Generalizes better (lower
variance - Occams razor)
Lecture Notes for E Alpaydn 2010 Introduc9on to Machine Learning 2e The MIT Press (V1.0)
Model Selection & Generalization
Learning is an ill-posed problem; data is not
sufficient to find a unique solution
Given d binary inputs, there are at most
2 d binary functions
d samples, and
2
2
Each sample eliminates half of the functions;
2d " N
Thus, N samples leaves 2
viable functions
!
Not possible to check all functions. Need for
inductive bias, assumptions about H
!
Generatlization and Overfitting
Generalization:
How well a model
performs on new data
Overfitting: H more complex than C or f
Underfitting: H less complex than C or f
Cross-Validation
Cross-Validation
To
better estimate generalization error, we
need data unseen during training. We split
the data as
Training
set (50%)
Validation set (25%)
Test set (25%)
Resampling
when there is few data
Lecture Notes for E Alpaydn 2010 Introduc9on to Machine Learning 2e The MIT Press (V1.0)
Triple Trade-Off
There is a trade-off between three factors
(Dietterich, 2003):
1.
2.
3.
Complexity of H, c (H),
Training set size, N,
Generalization error, E, on new data
As N, E
As c (H), first E and then E
Lecture Notes for E Alpaydn 2010 Introduc9on to Machine Learning 2e The MIT Press (V1.0)
Generalization Accuracy
Triple Trade-Off
400 examples
200 examples
100 examples
Complexity of Classier
Figure 2: Generalization accuracy as a function of the complexity of the classier, for various
amounts of training data.
Summary of Supervised Learning
1.
Model Selection:
2.
Loss function:
g ( x|! ) ! H
(
E ( h | X ) = #1 h ( x ) " r
!
t =1
t
t
)
Optimization procedure:
3.
(
E (" | X ) = # L r t , g( x t | " )
t
N
g( x ) = w1 x + w 0
!
!
)
N
1
E ( g | X ) = # r t " g( x t )
N t =1
[
" * = arg min E (" | X )
"
!
Algorithms: KNN, percepton, linear regression
!
]
2
CSCI 5525: Machine Learning (Spring 2012)
Bayes Decision
Theory and
Parametric Models
Rui Kuang
Department of Computer Science and Engineering
University of Minnesota
Probabilistic Perspective
We
have seen classification models.
Classification decision is deterministic
" 1 if h says x is positive
h( x) = #
$0 if h says x is negative
What
if we have cases that we are not so
sure about, such as data with outputs from
! stochastic process?
a
Estimation of
p(C = 0 | x ) and P (C = 1 | x )
Probability and Inference
Result
of tossing a coin is {Heads,Tails}
Random var X {1,0}
Bernoulli: P {X=1} = poX (1 po)(1 X)
Sample: X = {xt }Nt =1
Estimation:
po = # {Heads}/#{Tosses} = t xt / N
Prediction of next toss (no input):
Heads if po > , Tails otherwise
E. Alpaydin, Introduction to Machine Learning
Classification
Credit
scoring: Inputs are income Output and
savings. is low-risk vs high-risk
Input: x = [x1,x2]T ,Output: C is in {0,1}
Prediction:
"C = 1 if P (C = 1 | x1,x 2 ) > 0. 5
choose #
$C = 0 otherwise
or
"C = 1 if P (C = 1 | x1,x 2 ) > P (C = 0 | x1,x 2 )
choose #
$C = 0 otherwise
E. Alpaydin, Introduction to Machine Learning
Bayes Rule
How
to get P(C|x)?
prior
likelihood
posterior
P (C ) p( x | C )
P (C | x ) =
p( x )
evidence
P (C = 0) + P (C = 1) = 1
!
p( x ) = p( x | C = 1) P (C = 1) + p( x | C = 0) P (C = 0)
p(C = 0 | x ) + P (C = 1 | x ) = 1
E. Alpaydin, Introduction to Machine Learning
Bayes Rule Example
prior
likelihood
P (C ) p( x | C )
P (C | x ) =
p( x )
posterior
!
evidence
P (C = ' acc ') = 0.6, P (C = ' unacc ') = 0.4
Safty (x)
'high'
'low'
'med'
'high'
'low'
'med'
'high'
'low'
'med'
'high'
'low'
'med'
'high'
'low'
'med'
'high'
Rating (C)
'acc'
'unacc'
'acc'
'acc'
'unacc'
'acc'
'acc'
'unacc'
'unacc'
'acc'
'unacc'
'acc'
'acc'
'unacc'
'acc'
'acc'
Bayes Rule: K>2 Classes
p( x | Ci ) P (Ci )
P (C i | x ) =
p( x )
=
p( x | Ci ) P (Ci )
K
" p(x | C )P (C )
k
k
k =1
K
P (Ci ) " 0 and # P (Ci ) = 1
i =1
! choose C if P C | x = max P C | x
(i )
(k )
i
k
E. Alpaydin, Introduction to Machine Learning
!
Losses and Risks
Actions:
i
Loss of i when the state is Ck : ik
Expected risk (Duda and Hart, 1973)
K
R(" i | x ) = $ #ik P (Ck | x )
k =1
choose " i if R(" i | x ) = min k R(" k | x )
!
E. Alpaydin, Introduction to Machine Learning
Losses and Risks: 0/1 Loss
Loss
of i when the state is Ck : ik
K
$0 if i = k
"ik = %
&1 if i # k
!
R(" i | x ) = $ #ik P (Ck | x )
How likely the prediction Ci is wrong.
A soft cost based on the confidence
of a prediction.
k =1
= $ P (Ck | x )
k %i
= 1 & P (Ci | x )
For minimum risk, choose the most probable class
!
E. Alpaydin, Introduction to Machine Learning
Losses and Risks: Reject
#0 if i = k
%
"ik = $ " if i = K + 1, 0 < " < 1
%
&1 otherwise
R(" K +1 | x ) = #
!
R(" i | x ) = % P (Ck | x ) =1 & P (Ci | x )
k $i
choose Ci if P (Ci | x ) > P (Ck | x ) "k # i and P (Ci | x ) > 1 $ %
!
reject
otherwise
E. Alpaydin, Introduction to Machine Learning
!
Discriminant Functions
gi ( x ), i = 1,, K
$ "R(# | x )
i
&
gi ( x ) = % P (Ci | x )
&
' p( x | Ci ) P (Ci )
!
choose Ci if gi ( x ) = max k gk ( x )
!
!
K decision regions R1,...,RK
R i = {x | gi ( x ) = max k gk ( x )}
!
E. Alpaydin, Introduction to Machine Learning
K=2 Classes
Dichotomizer (K=2) vs Polychotomizer (K>2)
g(x) = g1(x) g2(x)
"C1 if g( x ) > 0
choose #
$C2 otherwise
Log odds:
!
!
log
P (C1 | x )
P (C2 | x )
E. Alpaydin, Introduction to Machine Learning
Parametric vs Nonparametric
Parametric methods:
A
model (usually a type of simple distribution) with a
few parameters (sufficient statistics) is assumed
Learning problem is to fit the model with the best
parameter to the data
The model is used for prediction
Nonparametric methods:
No
model/distribution is assumed
Predictions is made based on training instances
Semiparameteric
A
mix of model-based and instance-based learning