lec4-computational learning theory

lec4-computational learning theory - Machine Learning...

Info icon This preview shows pages 1–12. Sign up to view the full content.

View Full Document Right Arrow Icon
Machine Learning Lecture 4 Yang Yang Department of Computer Science & Engineering Shanghai Jiao Tong University
Image of page 1

Info icon This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Computational Learning Theory Reading: Andrew’s lecture notes Part VI Learning theory 《机器学习》 Chap. 12
Image of page 2
Generalizability of Learning In machine learning it s really the generalization error that we care about, but most learning algorithms fit their models to the training set. Why should doing well on the training set tell us anything about generalization error? Specifically, can we relate error on training set to generalization error? Are there conditions under which we can actually prove that learning algorithms will work well?
Image of page 3

Info icon This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
What General Laws Constrain Inductive Learning? Sample complexity How many training examples are sufficient to learn target concept? Computational complexity Resources required to learn target concept? What theory to relate: Training examples Quantity Quality m How presented Complexity of hypothesis / concept space H Accuracy of approx to target concept Probability of successful learning
Image of page 4
Two Basic Competing Models
Image of page 5

Info icon This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Protocol Given: set of examples X fixed (unknown) distribution D over X set of hypotheses H set of possible target concepts C Learner observes sample S = { } instances drawn from distribution D labeled by target concept c C Learner outputs h H estimating c h is evaluated by performance on subsequent instances drawn from D For now: C = H ( so c H ) noise-free data
Image of page 6
True Error of a Hypothesis Definition: The true error (denoted ) of hypothesis h with to target concept c and distribution D is the probability that h will misclassify an instance drawn at random according to D.
Image of page 7

Info icon This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Two Notions of Error Training error ( a.k.a., empirical risk or empirical error ) of hypothesis h with respect to target concept c How often h(x) ≠ c(x) over training instance from S True error of ( a.k.a., generalization error, test error ) hypothesis h with respect to c How often h(x) ≠ c(x) over future random instances drew i.i.d. from D
Image of page 8
The Union Bound Lemma. ( The union bound ). Let A 1 , A 2 , … , A k be k different events ( that may not be independent ). Then In probability theory, the union bound is usually stated as an axiom ( and thus we won t try to prove it ), but it also makes intuitive sense: The probability of any one of k events happening is at most the sums of the probabilities of the k different events.
Image of page 9

Info icon This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Hoeffding Inequality Lemma. ( Hoeffding Inequality ) Let Z 1 , … , Z m be m independent and identically distributed ( iid ) random variables drawn from a Bernoulli distribution, i.e., P( Z i = 1 ) = and P( Zi = 0 ) = 1 - Let be the mean of these random variables, and let any be fixed. Then This lemma ( which in learning theory is also called the Chernoff bound ) says that if we take — the average of m Bernoulli random variables — to be our estimate of , then the probability of our being far from the true value is small, so long as m is large.
Image of page 10
Version Space
Image of page 11

Info icon This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Image of page 12
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}

What students are saying

  • Left Quote Icon

    As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

    Student Picture

    Kiran Temple University Fox School of Business ‘17, Course Hero Intern

  • Left Quote Icon

    I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

    Student Picture

    Dana University of Pennsylvania ‘17, Course Hero Intern

  • Left Quote Icon

    The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

    Student Picture

    Jill Tulane University ‘16, Course Hero Intern