{[ promptMessage ]}

Bookmark it

{[ promptMessage ]}

9.1.Evaluating Hypotheses

# 9.1.Evaluating Hypotheses - Acknowledgement Material...

This preview shows pages 1–4. Sign up to view the full content.

11s1: COMP9417 Machine Learning and Data Mining Evaluating Hypotheses May 3, 2011 Acknowledgement: Material derived from slides for the book Machine Learning, Tom M. Mitchell, McGraw-Hill, 1997 http://www-2.cs.cmu.edu/~tom/mlbook.html and the book Data Mining, Ian H. Witten and Eibe Frank, Morgan Kaufmann, 2000. http://www.cs.waikato.ac.nz/ml/weka Aims This lecture will enable you to apply statistical and graphical methods to the evaluation of hypotheses in machine learning. Following it you should be able to: describe the problem of estimating hypothesis accuracy (error) define sample error and true error derive confidence intervals for observed hypothesis error understand learning algorithm comparisons using paired t -tests define and apply common evaluation measures generate lift charts and ROC curves COMP9417: May 3, 2011 Evaluating Hypotheses: Slide 1 [Recommended reading: Mitchell, Chapter 5] [Recommended exercises: 5.2 – 5.4] Relevant WEKA programs: weka.gui.experiment.Experimenter COMP9417: May 3, 2011 Evaluating Hypotheses: Slide 2

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
Evaluation in machine learning Machine learning is a highly empirical science . . . In theory, there is no di ff erence between theory and practice. But, in practice, there is. COMP9417: May 3, 2011 Evaluating Hypotheses: Slide 3 Estimating Hypothesis Accuracy how well does a hypothesis generalize beyond the training set ? need to estimate o ff -training-set error what is the probable error in this estimate ? if one hypothesis is more accurate than another on a data set, how probable is this di ff erence in general ? COMP9417: May 3, 2011 Evaluating Hypotheses: Slide 4 Two Definitions of Error The sample error of h with respect to target function f and data sample S is the proportion of examples h misclassifies error S ( h ) 1 n x S δ ( f ( x ) = h ( x )) Where δ ( f ( x ) = h ( x )) is 1 if f ( x ) = h ( x ) , and 0 otherwise ( cf. 0 1 loss). The true error of hypothesis h with respect to target function f and distribution D is the probability that h will misclassify an instance drawn at random according to D . error D ( h ) Pr x D [ f ( x ) = h ( x )] Question: How well does error S ( h ) estimate error D ( h ) ? COMP9417: May 3, 2011 Evaluating Hypotheses: Slide 5 Estimators Experiment: 1. choose sample S of size n according to distribution D 2. measure error S ( h ) error S ( h ) is a random variable (i.e., result of an experiment) error S ( h ) is an unbiased estimator for error D ( h ) Given observed error S ( h ) what can we conclude about error D ( h ) ? COMP9417: May 3, 2011 Evaluating Hypotheses: Slide 6
Problems Estimating Error 1. Bias: If S is training set, error S ( h ) is optimistically biased bias E [ error S ( h )] error D ( h ) For unbiased estimate, h and S must be chosen independently 2. Variance: Even with selection of S to give unbiased estimate, error S ( h ) may still vary from error D ( h ) COMP9417: May 3, 2011 Evaluating Hypotheses: Slide 7 Problems Estimating Error Note: Estimation bias not to be confused with Inductive bias – former is a numerical quantity [comes from statistics], latter is a set of assertions [comes from concept learning].

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}

### Page1 / 1

9.1.Evaluating Hypotheses - Acknowledgement Material...

This preview shows document pages 1 - 4. Sign up to view the full document.

View Full Document
Ask a homework question - tutors are online