lecture11 - Data Mining CS57300 Purdue University October 5...

Info icon This preview shows pages 1–7. Sign up to view the full content.

View Full Document Right Arrow Icon
Data Mining CS57300 Purdue University October 5, 2010
Image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Predictive modeling: evaluation
Image of page 2
Empirical evaluation Given observed accuracy of a model on limited data, how well does this estimate generalize for additional examples? Given that one model outperforms another on some sample of data, how likely is it that this model is more accurate in general? When data are limited, what is the best way to use the data to both learn and evaluate a model?
Image of page 3

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
How to evaluate a classifier? Use score function to assess quality of predictions for a set of instances Measures difference between the prediction of the model for an instance i and the true class label value of i Common functions: Zero-one loss Squared loss
Image of page 4
Evaluating classifiers Goal: Estimate true future error rate When data are limited, what is the best way to use the data to both learn and evaluate a model? Approach 1 Reclassify training data to estimate error rate
Image of page 5

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Approach 1 Y X1 X2 Data Set Model F(X) Y X1 X2 Data Set Score: 83% Typically produces a biased estimate of future error rate -- why?
Image of page 6
Image of page 7
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}

What students are saying

  • Left Quote Icon

    As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

    Student Picture

    Kiran Temple University Fox School of Business ‘17, Course Hero Intern

  • Left Quote Icon

    I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

    Student Picture

    Dana University of Pennsylvania ‘17, Course Hero Intern

  • Left Quote Icon

    The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

    Student Picture

    Jill Tulane University ‘16, Course Hero Intern