Tree Decision Learning CS 464: Introduction to Machine Learning
Decision Tree Learning
Slides adapted from Chapter 3 Machine Learning by Tom M. Mitchell http://www-2.cs.cmu.edu/afs/cs.cmu.edu/user/mitchell/ftp/mlbook.html 1
2
Decision tree learning is a method for approximating discrete-valued target functions. The learned function is represented by a decision tree. A learned decision tree can also be re-represented as a set of if-then rules. It is robust to noisy data and capable of learning disjunctive expressions. Decision tree learning is one of the most widely used and practical methods for inductive inference.
Decision Tree for PlayTennis
Outlook Sunny Humidity High No Normal Yes Overcast Rain
Decision Tree
Decision trees represent a disjunction of conjunctions of constraints on the attribute values of instances. Each path from the tree root to a leaf corresponds to a conjunction of attribute tests, and The tree itself is a disjunction of these conjunctions.
Yes
Wind Strong No Weak Yes
3
Outlook
(Outlook = Sunny Humidity = Normal) (Outlook = Overcast) (Outlook = Rain Wind = Weak)
Sunny Humidity Normal Yes
Overcast Rain Yes Wind Strong No
4
High No
Weak Yes
Decision Tree
Decision trees classify instances by sorting them down the tree from the root to some leaf node, which provides the classification of the instance. Each node in the tree specifies a test of some attribute of the instance. Each branch descending from a node corresponds to one of the possible values for the attribute. Each leaf node assigns a classification. The instance (Outlook=Sunny, Temperature=Hot, Humidity=High, Wind=Strong) is classified as a negative instance.
5
Properties of Decision Tree Learning
Continuous (real-valued) features can be handled by allowing nodes to split a real valued feature into two ranges based on a threshold (e.g. length < 3 and length 3) Classification trees have discrete class labels at the leaves, regression trees allow real-valued outputs at the leaves. Algorithms for finding consistent trees are efficient for processing large amounts of training data for data mining tasks. Methods developed for handling noisy training data (both class and feature noise). Methods developed for handling missing feature values.
6
Top-Down Decision Tree Induction (ID3)
Recursively build a tree top-down by divide and conquer. Main loop: 1. A the "best" decision attribute for next node 2. Assign A as decision attribute for node 3. For each value of A, create new descendant of node 4. Sort training examples to leaf nodes 5. If training examples perfectly classified, Then STOP, Else iterate over new leaf nodes
Which Attribute is Best?
S: [29+,35-] Attributes: A and B possible values for A: a,b possible values for B: c,d Entropy([29+,35-]) = -29/64 log2 29/64 35/64 log2 35/64 = 0.99 A? a [21+, 5-] b [8+, 30-] c [18+, 33-] B? d [11+, 2-]
7
8
Which Attribute is Best?
We want to select the attribute that is most useful for classifying examples. Information gain measures how well a given attribute separates the training examples according to their target classification. ID3 uses this information gain measure to select among the candidate attributes at each step while growing the tree. In order to define information gain precisely, we use a measure commonly used in information theory, called entropy Entropy characterizes the (im)purity of an arbitrary 9 collection of examples.
Picking a Good Split Attribute
Goal is to have the resulting tree be as small as possible, per Occam's razor. Finding a minimal decision tree (nodes, leaves, or depth) is an NP-hard optimization problem. Top-down divide-and-conquer method does a greedy search for a simple tree but does not guarantee to find the smallest. General lesson in ML: "Greed is good." We want to pick a feature that creates subsets of examples that are relatively "pure" in a single class so they are "closer" to being leaf nodes. There are a variety of heuristics for picking a good test, a popular one is based on information gain that originated with the ID3 system of Quinlan 10 (1979).
Measuring Entropy
S is a sample of training examples p is the proportion of positive examples in S p is the proportion of negative examples in S Entropy measures the impurity of S Entropy(S)=-p log p - p log p
11
Entropy Plot for Binary Classification
12
Entropy
If all examples are in one category, entropy is zero (we define 0 log(0) = 0) If examples are equally mixed (p=p=0.5), entropy is a maximum of 1. Entropy can be viewed as the number of bits required on average to encode the class of an example in S where data compression (e.g. Huffman coding) is used to give shorter codes to more likely cases. For multi-class problems with c categories, entropy generalizes to:
c
Information Gain
Gain(S, A) = expected reduction in entropy due to sorting S on A
Gain(S, A) Entropy(S) -
v in Values(A)
|Sv|/|S| Entropy(Sv)
Here, Sv is the set of training instances remaining from S after restricting to those for which attribute A has value v.
14
Entropy ( S ) = - pi log 2 ( pi )
i =1
13
Information Gain
Which Attribute is Best?
S: [29+,35-] Attributes: A and B possible values for A: a,b possible values for B: c,d Entropy([29+,35-]) = -29/64 log2 29/64 35/64 log2 35/64 = 0.99 A? a [21+, 5-] b [8+, 30-] c [18+, 33-] B? d [11+, 2-]
15
16
Which Attribute is Best?
E([29+,35-]) = 0.99 E([29+,35-]) = 0.99
Training Examples
Day Outlook D1 Sunny D2 Sunny D3 Overcast D4 Rain D5 Rain D6 Rain D7 Overcast D8 Sunny D9 Sunny D10 Rain D11 Sunny D12 Overcast D13 Overcast D14 Rain Temp Hot Hot Hot Mild Cool Cool Cool Mild Cool Mild Mild Mild Hot Mild Hum. High High High High Nml Nml Nml High Nml Nml Nml High Nml High Wind Weak Strong Weak Weak Weak Strong Strong Weak Weak Weak Strong Strong Weak Strong PlayTennis No No Yes Yes Yes No Yes No Yes Yes Yes Yes Yes No
A? a [21+, 5-]
E([21+,5-]) = 0.71
B? b [8+, 30-]
E([8+,30-]) = 0.74
c [18+, 33-]
d [11+, 2-]
E([18+,33-]) = 0.94 E([11+,2-]) = 0.62 Gain(S,B) = Entropy(S) -51/64*Entropy([18+,33-]) -13/64*Entropy([11+,2-]) = 0.12
17
Gain(S,A) = Entropy(S) -26/64*Entropy([21+,5-]) -38/64*Entropy([8+,30-]) = 0.27
A provides greater information gain than B. A is a better classifier than B.
18
ID3 Selecting ext Attribute
Entropy([9+,5-] = (9/14) log2(9/14) (5/14) log2(5/14) = 0.940 S=[9+,5-] E=0.940 Humidity High [3+, 4-] E=0.985 Normal [6+, 1-] E=0.592
Gain(S,Wind) = 0.940-(8/14)*0.811(6/14)*1.0 = 0.048
ID3 Selecting ext Attribute
S=[9+,5-] E=0.940 Outlook Sunny Overcast [4+, 0] E=0.0 Rain [3+, 2-] E=0.971
S=[9+,5-] E=0.940 Wind Weak [6+, 2-] Strong [3+, 3-]
[2+, 3-] E=0.971
Gain(S,Humidity) = 0.940-(7/14)*0.985-(7/14)*0.592 = 0.151
Gain(S,Outlook) = 0.940-(5/14)*0.971 -(4/14)*0.0 -(5/14)*0.0971 = 0.247
19 20
ID3 Selecting ext Attribute
S=[9+,5-] E=0.940 Temp. Hot [2+, 2-] E=1.0 Mild [4+, 2-] E= 0.911 Cold [3+, 1-] E=0.811
ID3 - Best Attribute: Outlook
S=[9+,5-] S={D1,D2,...,D14} Outlook Sunny [2+, 3-]
Ssunny={D1,D2,D8,D9,D11}
Overcast [4+, 0]
Rain [3+, 2-]
Srain={D4,D5,D6,D10,D14}
Sovercast={D3,D7,D12,D13}
?
Yes
?
Gain(S,Outlook) = 0.940-(4/14)*1.0 - (6/14)*0.911 - (4/14)*0.811 = 0.029
21
Which attribute should be tested here?
22
Attribute Bottom Left?
S=[9+,5-] S={D1,D2,...,D14} Outlook Sunny [2+, 3-]
Ssunny={D1,D2,D8,D9,D11}
Comparing Attributes for Ssunny
Ssunny = {D1,D2,D8,D9,D11} Gain (Ssunny , Humidity) = .970 - (3/5) 0.0 - (2/5) 0.0 = .970 Gain (Ssunny , Temp)
Overcast [4+, 0]
Rain [3+, 2-]
Srain={D4,D5,D6,D10,D14}
Sovercast={D3,D7,D12,D13}
?
Yes
?
= .970 - (2/5) 0.0 - (2/5) 1.0 - (1/5) 0.0 = .570 Gain (Ssunny , Wind) = .970 - (2/5) 1.0 - (3/5) .918 = .019 Humidity will be selected.
23
24
ID3 - Result
Outlook Sunny Humidity High No [D1,D2,D8] Normal Yes [D9,D11] Overcast Yes [D3,D7,D12,D13] Rain Wind Strong No [D6,D14] Weak Yes [D4,D5,D10]
25
ID3 - Algorithm
ID3(Examples, TargetAttribute, Attributes) Create a Root node for the tree If all Examples are positive, Return the single-node tree Root, with label = + If all Examples are negative, Return the single-node tree Root, with label = If Attributes is empty, Return the single-node tree Root, with label = most common value of TargetAttribute in Examples Otherwise Begin A the attribute from Attributes that best classifies Examples The decision attribute for Root A For each possible value, vi, of A, Add a new tree branch below Root, corresponding to the test A = vi Let Examplesvi be the subset of Examples that have value vi for A If Examplesvi is empty Then below this new branch add a leaf node with label = most common value of TargetAttribute in Examples Else below this new branch add the subtree ID3(Examplesvi , TargetAttribute, {A}) Attributes 26 Return Root
Hypothesis Space Search by ID3
Performs batch learning that processes all training instances at once rather than incremental learning that updates a hypothesis after each example. Performs hill-climbing (greedy search) that may only find a locally-optimal solution. Guaranteed to find a tree consistent with any conflict-free training set (i.e. identical feature vectors always assigned the same class), but not necessarily the simplest tree. Finds a single discrete hypothesis, so there is no way to provide confidences or create useful queries.
27
Hypothesis Space Search by ID3
ID3: representation: trees Scoring: entropy Search: greedy
28
Hypothesis Space Search by ID3
Hypothesis space is complete!
Target function surely in there...
Inductive Bias in ID3
Note H is the power set of instances X Unbiased? Not really... Preference for 1) short trees, and 2) trees with high information gain attributes near the root Bias is a preference for some hypotheses, rather than a restriction of hypothesis space H Occam's razor: prefer the shortest hypothesis that fits the data 30
Outputs a single hypothesis (which one?)
Can't play 20 questions...
No backtracking
Local minima...
Statistically-based search choices
Robust to noisy data...
Inductive bias "prefer shortest tree"
29
Occam's Razor
Why prefer short hypotheses? Argument in favor: Fewer short hypothesis than long ones. a short hypothesis that fits data unlikely to be coincidence a long hypothesis that fits data might be coincidence Argument opposed: There are many ways to define small sets of hypotheses. What's so special about small sets based on size of hypothesis?
31
Inductive Bias in ID3 Restriction Bias and Preference Bias
ID3 searches a complete hypothesis space
It searches incompletely through this space, from simple to complex hypotheses, until its termination condition is met. Its inductive bias is solely a consequence of the ordering of hypotheses by its search strategy. Its hypothesis space introduces no additional bias.
Candidate Elimanation searches an incomplete hypothesis space
It searches this space completely, finding every hypothesis consistent with the training data. Its inductive bias is solely a consequence of the expressive power of its hypothesis representation. Its search strategy introduces no additional bias.
32
Inductive Bias in ID3 Restriction Bias and Preference Bias
The inductive bias of ID3 is a preference for certain hypotheses over others, with no hard restriction on the hypotheses that can be eventually enumerated. This is called preference bias (search bias). The inductive bias of Candidate Elimination is in the form of a categorical restriction on the set of hypotheses considered. This is called restriction bias (language bias). A preference bias is more desirable than a restriction bias, because it allows the learner to work within a complete hypothesis space that is assured to contain the unknown target function.
33
Overfitting
Consider adding noisy training example #15: Sunny, Hot, ormal, Strong, PlayTennis = o What effect on earlier tree?
ID3 will sort it into 2nd leaf node which will incorrectly classify it. This will cause ID3 to search for refinements to the tree below 34 this node.
Overfitting
Consider error of hypothesis h over training data: errortrain(h) entire distribution D of data: errorD(h) Hypothesis h in H overfits training data if there is an alternative hypothesis h' in H such that errortrain(h) < errortrain(h'), and errorD(h) > errorD(h')
35
Reasons for Overfitting
Errors and noise in training examples Coincidental regularities, especially small number of examples associated with leaf nodes
36
Overfitting in Learning
Overfitting in Learning
As ID3 adds new nodes to grow the decision tree, the accuracy of the tree 37 measured over the training examples increases monotonically.
However, when measured over a set of test examples independent of the training
examples, accuracy first increases, then decreases.
38
Avoiding Overfitting
How can we avoid overfitting? stop growing when data split not statistically significant grow full, then post-prune How to select "best" tree: Measure performance over training data Measure performance over separate validation data set MDL: minimize size(tree) + size(misclassifications(tree))
39
Reduced-Error Pruning
Split data into training and validation set Do until further pruning is harmful: 1. Evaluate impact on validation set of pruning each possible node (plus those below it) 2. Greedily remove the one that most improves validation set accuracy produces smallest version of most accurate subtree What if data is limited? 40
Effect of Pruning
Rule-Post Pruning
Another sucessful method for finding high accuracy hypotheses. Used by C4.5 algorithm (an extension of ID3). Steps of Rule-Post Pruning:
Infer the decision tree from the training set. Convert the learned tree into an equivalent set of rules by creating one rule for each path from the root node to a leaf node. Prune (generalize) each rule by removing any preconditions that result in improving its estimated accuracy. Sort the pruned rules by their estimated accuracy, and consider 42 them in this sequence when classifying subsequent instances.
41
Converting Tree to Rules
Outlook Sunny Overcast Rain Humidity High No Normal Yes Yes Wind Strong No Weak Yes
Pruning Rules
Each rule is pruned by removing any antecedent (precondition).
Ex. Prune R1 by removing (Outlook=Sunny) or (Humidity=High)
Select whichever of the pruning steps produced the greatest improvement in estimated rule accuracy.
Then, continue with other preconditions. No pruning step is performed if it reduces the estimated rule accuracy.
R1: If (Outlook=Sunny) (Humidity=High) Then PlayTennis=No R2: If (Outlook=Sunny) (Humidity=Normal) Then PlayTennis=Yes R3: If (Outlook=Overcast) Then PlayTennis=Yes R4: If (Outlook=Rain) (Wind=Strong) Then PlayTennis=No 43 R5: If (Outlook=Rain) (Wind=Weak) Then PlayTennis=Yes
In order to estimate rule accuracy:
use a validation set of examples disjoint from the training set evaluate performance based on the training set itself(using statistical 44 techniques). C4.5 uses this approach.
Why Convert The Decision Tree To Rules Before Pruning?
Converting to rules improves readability.
Rules are often easier to understand.
Cost-complexity pruning
C4.5's postpruning often does not prune enough
Tree size continues to grow when more instances are added even if performance on independent data does not improve Very fast and popular in practice
Distinguishing different contexts in which a node is used
separate pruning decision for each path
No difference for root/inner
no bookkeeping on how to reorganize tree if root node is pruned
Can be worthwhile in some cases to strive for a more compact tree
At the expense of more computational effort Cost-complexity pruning method from the CART (Classification and Regression Trees) learning system
46
45
Cost-complexity pruning
Basic idea:
First prune subtrees that, relative to their size, lead to the smallest increase in error on the training data Increase in error () average error increase per leaf of subtree Pruning generates a sequence of successively smaller trees
Each candidate tree in the sequence corresponds to one particular threshold value,
Continuous Valued Attributes
Create a discrete attribute to test continuous Temp = 82.5 (Temp > 72.3) = t, f
Temp: 40 48 60 72 80 90 PlayTennis: No No Yes Yes Yes No
Which tree to chose as the final model?
47
48
Use either a hold-out set or cross-validation to estimate the error of each
Applications for Decision Trees?
Instances describable by attribute-value pairs Target function is discrete valued Disjunctive hypothesis may be required Possibly noisy training data
Summary
Decision tree learning provides a practical method for learning discrete-valued functions.
decision trees are inferred by growing them from the root downward, greedily selecting the next best attribute.
Examples: Equipment or medical diagnosis Credit risk analysis Target marketing
ID3 searches a complete hypothesis space. The inductive bias in ID3 includes a preference for smaller trees. Overfitting training data is an important issue in decision tree learning. Pruning decision trees or rules are important. A large variety of extensions to the basic ID3 algorithm has been developed. These extensions include methods for
post-pruning trees, handling real-valued attributes, handling unknown attribute values, using attribute selection measures other than information gain, and considering costs associated with instance attributes.
50
49