Assignment Homework 4 Solution (CS 171, Winter, 2008)
Graded on a 10 point scale. Problem 1 a, worth 1 point. .5 point for a correctly structured tree. .5 point for correctly classifying all examples. Problem 1 b, worth 5 points. Points were given out for correct calculations and choices: 1 point for initial information value 1 point for outlook on the first level .5 points for humidity calculation on the first level .5 points for windy calculation on the first level .5 points for temperature calculation on the first level .5 points for humidity calculation on the second level .5 points for windy calculation on the second level .5 points for making the correct classifications For constructing a decision tree in general using the given algorithm, we want to split at each node based on the attribute which gives us the greatest information gain. We then continue to split nodes until each leaf node contains only one class. The information gain for splitting on attribute A with 3 possible values is defined by: Info. Gain(A) = Info(old) - (prob. v1 * Info(v1) + prob. v2 * Info(v2) + prob. v3 *Info(v3)) Where the Info is defined by the equation: Info(c1, c2) = -P(c1)*log P(c1) - P(c2)*log P(c2) Where P(c1) is the probability of class 1, and all logarithms are base two. So our first task is to find which attribute to split our root node on. We begin by calculating initial information content Info for our data set: Info (p, n) = -P(p)*logP(p) - P(n)*logP(n) = -(9/14)*log(9/14) - (5/14)*log(5/14) = 0.41 + 0.53 = 0.94
This value is henceforth referred to as Info(old). We next calculate the information content of each of the child nodes after splitting on each of the possible attributes (outlook, temperature, humidity, wind). For outlook: Info(sunny) = I(2/5,3/5) = -(2/5)*log(2/5) - (3/5)*log(3/5) = 0.44 + .53 .97 Info(rain) = I(3/5,2/5) = .44 + .53 = .97 Info(overcast) = I(4/4,0) =0+0 =0 Info. Gain (outlook) = Info(old) -(5/14*Info(sunny) + 5/14*Info(rain) + 4/14*Info(Overcast)) = .94 -(.346 + .346 + 0) = .25
For temperature: Info(hot) = I(2/4,2/4) =1 Info (mild) = I(4/6, 2/6) = .39 + .53 = .92 Info (cool) = I(3/4, 1/4) = .31 + .5 = .81
Info. Gain (temperature) = Info(old) -(4/14*Info(hot) + 6/14*Info(cool) + 4/14*Info(Mild)) = .94 -(.29 + .39 + .23) = .02 For Humidity: Info (high) = I(3/7, 4/7) = .98 Info (normal) = I(6/7, 1/7) = .59 Info. Gain (Humidity) = Info(old) -(7/14*Info(high) + 7/14*Info(normal) = .94 - .49 -.29 = .16 For Windy: Info (true) = I(3/6, 3/6) =1 Info (False) = I(6/8, 2/8) = .81 Info. Gain (Windy) = Info(old) -(6/14*Info(true) + 8/14*Info(false) = .94 .43 - - .46 = .05 So we choose to split on Outlook, which has the highest Info gain.
So our tree now looks like this:
Outlook Sunny P:2 N:3 OverCast P:4 N:0 Rain P:2 N:3
Now we need to expand each of the subnodes with the same process, calculating based on the instances within the subnode. Since the Overcast node already has only one class, we don't need to expand it any further. For the Sunny node: Info(Sunny) = .97 splitting on Windy: Info (true) = I(1/2, 1/2) =1 Info (false) = I(1/3, 2/3) = .92 Info. Gain (Windy) = Info(sunny) - (2/5*Info(true) + 3/5*Info(false) = .97 -.4 - .55 = .02
splitting on Humidity: Info (high) = I(0, 3/3) =0 Info (normal) = I(2/2, 0) =0 Info. Gain (Humidity) = Info(sunny) - (3/5*Info(high) + 2/5*Info(normal) = .97 - 0 - 0 = .97 Because this split of the classes is the best possible, we can choose humidity as our attribute, without calculating temperature. A similar calculation on the rainy node will find that the windy will split the same way. Our final tree will then look like:
Outlook Sunny P:2 N:3 Humidity High P:0 N:3 N Y Normal P:2 N:0 OverCast P:4 N:0 Y True P:0 N:3 N Rain P:2 N:3 Windy False P:2 N:0 Y
Problem 2, worth 2 points. Sample answer: The reason a node is not revisited is all records that have passed through a given node once are homogenous in regards to the result of the test performed by that node and hence will not be divided with respect to another test performed by the same node.
Problem 3, worth 2 points. Following the same notation in the book, we get the following. If training set con...