L11testing - Testing classifier accuracy (cse352) Professor...

Info iconThis preview shows pages 1–9. Sign up to view the full content.

View Full Document Right Arrow Icon
Testing classifier accuracy (cse352) Professor Anita Wasilewska Lecture Notes on Learning (6)
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Overview • Introduction • Basic Concept on Training and Testing Resubstitution (N ; N) Holdout (2N/3 ; N/3) x-fold cross-validation (N-N/x ; N/x) Leave-one-out (N-1 ; 1)
Background image of page 2
Introduction Predictive Accuracy Evaluation The main methods of predictive accuracy evaluations are: Resubstitution (N ; N) Holdout (2N/3 ; N/3) x-fold cross-validation (N-N/x ; N/x) Leave-one-out (N-1 ; 1) where N is the number of records (instances) in the dataset
Background image of page 3

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Training and Testing REMEMBER: we must know the classification (class attribute values) of all instances (records) used in the test procedure. Basic Concepts Success: instance (record) class is predicted correctly Error: instance class is predicted incorrectly Error rate : a percentage of errors made over the whole set of instances (records) used for testing Predictive Accuracy: a percentage of well classified data in the testing data set.
Background image of page 4
Training and Testing • Example: Testing Rules (testing record #1) = record #1.class - Succ Testing Rules (testing record #2) not= record #2.class - Error Testing Rules (testing record #3) = record #3.class - Succ Testing Rules (testing record #4) = instance #4.class - Succ Testing Rules (testing record #5) not= record #5.class - Error Error rate: 2 errors: #2 and #5 Error rate = 2/5=40% Predictive Accuracy: 3/5 = 60%
Background image of page 5

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Resubstitution (N ; N)
Background image of page 6
Re-substitution Error Rate • Re-substitution error rate is obtained from training data • Training Data Error: uncertainty of the rules • The error rate is not always 0%, but usually (and hopefully) very low! • Resubstitution error rate indicates only how good (bad ) are our results (rules, patterns, NN) on the TRAINING data; expresses some knowledge about the algorithm used.
Background image of page 7

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Re-substitution Error Rate Re-substitution Error Rate is usually used as the performance measure: The training error rate reflects imprecision of the training
Background image of page 8
Image of page 9
This is the end of the preview. Sign up to access the rest of the document.

This note was uploaded on 01/25/2012 for the course CSE 352 taught by Professor Wasilewska,a during the Fall '08 term at SUNY Stony Brook.

Page1 / 24

L11testing - Testing classifier accuracy (cse352) Professor...

This preview shows document pages 1 - 9. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online