lec4-computational learning theory

# lec4-computational learning theory - Machine Learning...

This preview shows pages 1–12. Sign up to view the full content.

Machine Learning Lecture 4 Yang Yang Department of Computer Science & Engineering Shanghai Jiao Tong University

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
Computational Learning Theory Reading: Andrew’s lecture notes Part VI Learning theory 《机器学习》 Chap. 12
Generalizability of Learning In machine learning it s really the generalization error that we care about, but most learning algorithms fit their models to the training set. Why should doing well on the training set tell us anything about generalization error? Specifically, can we relate error on training set to generalization error? Are there conditions under which we can actually prove that learning algorithms will work well?

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
What General Laws Constrain Inductive Learning? Sample complexity How many training examples are sufficient to learn target concept? Computational complexity Resources required to learn target concept? What theory to relate: Training examples Quantity Quality m How presented Complexity of hypothesis / concept space H Accuracy of approx to target concept Probability of successful learning
Two Basic Competing Models

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
Protocol Given: set of examples X fixed (unknown) distribution D over X set of hypotheses H set of possible target concepts C Learner observes sample S = { } instances drawn from distribution D labeled by target concept c C Learner outputs h H estimating c h is evaluated by performance on subsequent instances drawn from D For now: C = H ( so c H ) noise-free data
True Error of a Hypothesis Definition: The true error (denoted ) of hypothesis h with to target concept c and distribution D is the probability that h will misclassify an instance drawn at random according to D.

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
Two Notions of Error Training error ( a.k.a., empirical risk or empirical error ) of hypothesis h with respect to target concept c How often h(x) ≠ c(x) over training instance from S True error of ( a.k.a., generalization error, test error ) hypothesis h with respect to c How often h(x) ≠ c(x) over future random instances drew i.i.d. from D
The Union Bound Lemma. ( The union bound ). Let A 1 , A 2 , … , A k be k different events ( that may not be independent ). Then In probability theory, the union bound is usually stated as an axiom ( and thus we won t try to prove it ), but it also makes intuitive sense: The probability of any one of k events happening is at most the sums of the probabilities of the k different events.

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
Hoeffding Inequality Lemma. ( Hoeffding Inequality ) Let Z 1 , … , Z m be m independent and identically distributed ( iid ) random variables drawn from a Bernoulli distribution, i.e., P( Z i = 1 ) = and P( Zi = 0 ) = 1 - Let be the mean of these random variables, and let any be fixed. Then This lemma ( which in learning theory is also called the Chernoff bound ) says that if we take — the average of m Bernoulli random variables — to be our estimate of , then the probability of our being far from the true value is small, so long as m is large.
Version Space

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}

### What students are saying

• As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

Kiran Temple University Fox School of Business ‘17, Course Hero Intern

• I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

Dana University of Pennsylvania ‘17, Course Hero Intern

• The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

Jill Tulane University ‘16, Course Hero Intern