{[ promptMessage ]}

Bookmark it

{[ promptMessage ]}

lect15

# lect15 - Model Evaluation • Metrics for Performance...

This preview shows pages 1–9. Sign up to view the full content.

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: Model Evaluation • Metrics for Performance Evaluation – How to evaluate the performance of a model? • Methods for Performance Evaluation – How to obtain reliable estimates? • Methods for Model Comparison – How to compare the relative performance of different models? Metrics for Performance Evaluation • Focus on the predictive capability of a model – Rather than how fast it takes to classify or build models, scalability, etc. • Confusion Matrix: PREDICTED CLASS ACTUAL CLASS Class=Yes Class=No Class=Yes a: TP b: FN Class=No c: FP d: TN a: TP (true positive) b: FN (false negative) c: FP (false positive) d: TN (true negative) Metrics for Performance Evaluation… • Most widely-used metric: PREDICTED CLASS ACTUAL CLASS Class=Yes Class=No Class=Yes a (TP) b (FN) Class=No c (FP) d (TN) FN FP TN TP TN TP d c b a d a + + + + = + + + + = Accuracy Limitation of Accuracy • Consider a 2-class problem – Number of Class 0 examples = 9990 – Number of Class 1 examples = 10 • If model predicts everything to be class 0, accuracy is 9990/10000 = 99.9 % – Accuracy is misleading because model does not detect any class 1 example Cost Matrix PREDICTED CLASS ACTUAL CLASS C(i|j) Class=Yes Class=No Class=Yes C(Yes|Yes) C(No|Yes) Class=No C(Yes|No) C(No|No) C(i|j): Cost of misclassifying class j example as class i Computing Cost of Classification Cost Matrix PREDICTED CLASS ACTUAL CLASS C(i|j) +- +-1 100- 1 Model M 1 PREDICTED CLASS ACTUAL CLASS +- + 150 40- 60 250 Model M 2 PREDICTED CLASS ACTUAL CLASS +- + 250 45- 5 200 Accuracy = 80% Cost = 3910 Accuracy = 90% Cost = 4255 Cost vs Accuracy Count PREDICTED CLASS ACTUAL CLASS Class=Yes Class=No Class=Yes a b Class=No c d Cost PREDICTED CLASS ACTUAL CLASS Class=Yes Class=No Class=Yes p q Class=No q p Accuracy is proportional to cost if 1. C(Yes|No)=C(No|Yes) = q 2. C(Yes|Yes)=C(No|No) = p N = a + b + c + d Accuracy = (a + d)/N Cost = p (a + d) + q (b + c) = p (a + d) + q (N – a – d) = q N – (q – p)(a + d) = N [q – (q-p) × Accuracy] Cost-Sensitive Measures FN FP TP TP c b a a p r rp FN TP TP b a...
View Full Document

{[ snackBarMessage ]}

### Page1 / 31

lect15 - Model Evaluation • Metrics for Performance...

This preview shows document pages 1 - 9. Sign up to view the full document.

View Full Document
Ask a homework question - tutors are online