lect15

lect15 - Model Evaluation Metrics for Performance...

Info iconThis preview shows pages 1–9. Sign up to view the full content.

View Full Document Right Arrow Icon

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: Model Evaluation Metrics for Performance Evaluation How to evaluate the performance of a model? Methods for Performance Evaluation How to obtain reliable estimates? Methods for Model Comparison How to compare the relative performance of different models? Metrics for Performance Evaluation Focus on the predictive capability of a model Rather than how fast it takes to classify or build models, scalability, etc. Confusion Matrix: PREDICTED CLASS ACTUAL CLASS Class=Yes Class=No Class=Yes a: TP b: FN Class=No c: FP d: TN a: TP (true positive) b: FN (false negative) c: FP (false positive) d: TN (true negative) Metrics for Performance Evaluation Most widely-used metric: PREDICTED CLASS ACTUAL CLASS Class=Yes Class=No Class=Yes a (TP) b (FN) Class=No c (FP) d (TN) FN FP TN TP TN TP d c b a d a + + + + = + + + + = Accuracy Limitation of Accuracy Consider a 2-class problem Number of Class 0 examples = 9990 Number of Class 1 examples = 10 If model predicts everything to be class 0, accuracy is 9990/10000 = 99.9 % Accuracy is misleading because model does not detect any class 1 example Cost Matrix PREDICTED CLASS ACTUAL CLASS C(i|j) Class=Yes Class=No Class=Yes C(Yes|Yes) C(No|Yes) Class=No C(Yes|No) C(No|No) C(i|j): Cost of misclassifying class j example as class i Computing Cost of Classification Cost Matrix PREDICTED CLASS ACTUAL CLASS C(i|j) +- +-1 100- 1 Model M 1 PREDICTED CLASS ACTUAL CLASS +- + 150 40- 60 250 Model M 2 PREDICTED CLASS ACTUAL CLASS +- + 250 45- 5 200 Accuracy = 80% Cost = 3910 Accuracy = 90% Cost = 4255 Cost vs Accuracy Count PREDICTED CLASS ACTUAL CLASS Class=Yes Class=No Class=Yes a b Class=No c d Cost PREDICTED CLASS ACTUAL CLASS Class=Yes Class=No Class=Yes p q Class=No q p Accuracy is proportional to cost if 1. C(Yes|No)=C(No|Yes) = q 2. C(Yes|Yes)=C(No|No) = p N = a + b + c + d Accuracy = (a + d)/N Cost = p (a + d) + q (b + c) = p (a + d) + q (N a d) = q N (q p)(a + d) = N [q (q-p) Accuracy] Cost-Sensitive Measures FN FP TP TP c b a a p r rp FN TP TP b a...
View Full Document

This document was uploaded on 10/05/2010.

Page1 / 31

lect15 - Model Evaluation Metrics for Performance...

This preview shows document pages 1 - 9. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online