1
© Eric Xing @ CMU, 2006-2008
1
Machine Learning
Machine Learning
10
10-
701/15
701/15-
781, Fall 2008
781, Fall 2008
Computational Learning Theory
Computational Learning Theory
Eric Xing
Eric Xing
Lecture 10, October 8, 2008
Reading: Chap. 7 T.M book
© Eric Xing @ CMU, 2006-2008
2
Generalizability of Learning
z
In machine learning it's really generalization error that we care
about, but most learning algorithms fit their models to the
training set.
z
Why should doing well on the training set tell us anything
about generalization error? Specifically, can we relate error on
to training set to generalization error?
z
Are there conditions under which we can actually prove that
learning algorithms will work well?

This
** preview**
has intentionally

**sections.**

*blurred***to view the full version.**

*Sign up*
2
© Eric Xing @ CMU, 2006-2008
3
Complexity of Learning
z
The complexity of leaning is measured mainly along
two axis:
Information
Information
and
computation
.
The
Information complexity
is concerned with the
generalization performance of learning;
z
How many training examples are needed?
z
How fast do learner’s estimate converge to the true population parameters? etc.
The
Computational complexity
concerns the computation
resources applied to the training data to extract from it
learner’s predictions.
It seems that when an algorithm improves with respect to one of
these measures it deteriorates with respect to the other.
© Eric Xing @ CMU, 2006-2008
4
What General Laws
constrain Inductive Learning?
These results only useful wrt O(…) !
z
Sample Complexity
z
How many training examples are sufficient
to learn target concept?
z
Computational Complexity
z
Resources required to learn target concept?
z
Want theory to relate:
z
Training examples
z
Quantity
z
Quality
m
z
How presented
z
Complexity of hypothesis/concept space
H
z
Accuracy of approx to target concept
ε
z
Probability of successful learning
δ