learn a complete and consistent decision tree that classifies all examples in

Learn a complete and consistent decision tree that

This preview shows page 39 - 47 out of 59 pages.

learn a complete and consistent decision tree that classifies all examples in the training set correctly 2. as long as the performance increases try simplification operators on the tree evaluate the resulting trees make the replacement the results in the best estimated performance 3. return the resulting decision tree
Image of page 39

Subscribe to view the full document.

Postpruning 4 0 © J. Fürnkranz Two subtree simplification operators Subtree replacement Subtree raising Possible performance evaluation strategies error estimation on separate pruning set („reduced error pruning“) with confidence intervals (C4.5's method) significance testing MDL principle
Image of page 40
Subtree replacement Bottom-up Consider replacing a tree only after considering all its subtrees 4 1 © J. Fürnkranz
Image of page 41

Subscribe to view the full document.

Subtree raising 4 2 © J. Fürnkranz Delete node B Redistribute instances of leaves 4 and 5 into C
Image of page 42
Estimating Error Rates 4 3 © J. Fürnkranz Prune only if it does not increase the estimated error Error on the training data is NOT a useful estimator (would result in almost no pruning) Reduced Error Pruning Use hold-out set for pruning Essentially the same as in rule learning only pruning operators differ (subtree replacement) C4.5’s method Derive confidence interval from training data with a user-provided confidence level Assume that the true error is on the upper bound of this confidence interval (pessimistic error estimate)
Image of page 43

Subscribe to view the full document.

Pessimistic Error Rates Consider classifying E examples incorrectly out of N examples as observing E events in N trials in the binomial distribution. For a given confidence level CF, the upper limit on the error rate over the whole population is U CF ( E , N ) with CF% confidence. Example: 100 examples in a leaf 6 examples misclassified How large is the true error assuming a pessimistic estimate with a confidence of 25%? Note: this is only a heuristic! but one that works well L 0.25 (100,6) 6 Possibility(%) 2 10 U 0.25 (100,6 ) 75% confidence interval 4 4 © J. Fürnkranz
Image of page 44
Reduced Error Pruning 4 5 © J. Fürnkranz basic idea optimize the accuracy of a decision tree on a separate pruning set 1. split training data into a growing and a pruning set 2. learn a complete and consistent decision tree that classifies all examples in the growing set correctly 3. as long as the error on the pruning set does not increase try to replace each node by a leaf (predicting the majority class) evaluate the resulting (sub-)tree on the pruning set make the replacement the results in the maximum error reduction 4. return the resulting decision tree
Image of page 45

Subscribe to view the full document.

Complexity of tree induction 4 6 © J. Fürnkranz Assume m attributes n training instances tree depth O (log n ) Building a tree Subtree replacement Subtree raising O ( m n log n ) O ( n ) O ( n (log n ) 2 ) Every instance may have to be redistributed at every node between its leaf and the root Cost for redistribution (on average): O (log n ) Total cost: O ( m n log n ) + O ( n (log n ) 2 )
Image of page 46
Image of page 47
  • Fall '17
  • Mark Isbel

What students are saying

  • Left Quote Icon

    As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

    Student Picture

    Kiran Temple University Fox School of Business ‘17, Course Hero Intern

  • Left Quote Icon

    I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

    Student Picture

    Dana University of Pennsylvania ‘17, Course Hero Intern

  • Left Quote Icon

    The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

    Student Picture

    Jill Tulane University ‘16, Course Hero Intern

Ask Expert Tutors You can ask 0 bonus questions You can ask 0 questions (0 expire soon) You can ask 0 questions (will expire )
Answers in as fast as 15 minutes