View the step-by-step solution to:

Decision Tree Learning Page 1 of 3 11.3 A worked example We will stick with our weekend example. Suppose we want to train a decision tree using the...

One major issue for any decision tree algorithm is how to choose an attribute based on which the data set can be categorized and a well-balanced tree can be created. The most traditional approach is called the ID3 algorithm proposed by Quinlan in 1986. The detailed ID3 algorithm is shown in the slides. The textbook provides some discussions on the algorithm in Section 18.3. For this problem please follow the ID3 algorithm and manually calculate the values based on a data set similar to (but not the same as) the one in the slides (p. 147). This exercise should help you get deep insights on the execution of the ID3 algorithm. Please note that concepts discussed here (for example, entropy, information gain) are very important in information theory and signal processing fields. The new data set is shown as follows. In this example row 10 is removed from the original set and all other rows remain the same.

Following the conventions used in the slides, please show a manual process and calculate the following values: Entropy(S), Entropy(S weather = sunny), Entropy(S weather = windy), Entropy(S weather = rainy), Gain (S, weather), Gain (S, parents) and Gain (S, money). Based on the last three values, which attribute should be chosen to split on?


The file I have included is the same needed information on the slides but much better quality for reading. I reallly need to understand how to do this for other questions, so I would appreciate it if you could show how you do it too.

11.3 A worked example We will stick with our weekend example. Suppose we want to train a decision tree using the following instances: The first thing we need to do is work out which attribute will be put into the node at the top of our tree: either weather, parents or money. To do this, we need to calculate: Entropy(S) = -p cinema log 2 (p cinema ) -p tennis log 2 (p tennis ) -p shopping log 2 (p shopping ) -p stay_in log 2 (p stay_in ) = -(6/10) * log 2 (6/10) -(2/10) * log 2 (2/10) -(1/10) * log 2 (1/10) -(1/10) * log 2 (1/10) = -(6/10) * -0.737 -(2/10) * -2.322 -(1/10) * -3.322 -(1/10) * -3.322 = 0.4422 + 0.4644 + 0.3322 + 0.3322 = 1.571 and we need to determine the best of: Gain(S, weather) = 1.571 - (|S sun |/10)*Entropy(S sun ) - (|S wind |/10)*Entropy(S wind ) - (|S rain |/10)*Entropy(S rain ) = 1.571 - (0.3)*Entropy(S sun ) - (0.4)*Entropy(S wind ) - (0.3)*Entropy(S rain ) = 1.571 - (0.3)*(0.918) - (0.4)*(0.81125) - (0.3)*(0.918) = 0.70 Gain(S, parents) = 1.571 - (|S yes |/10)*Entropy(S yes ) - (|S no |/10)*Entropy(S no ) = 1.571 - (0.5) * 0 - (0.5) * 1.922 = 1.571 - 0.961 = 0.61 Gain(S, money) = 1.571 - (|S rich |/10)*Entropy(S rich ) - (|S poor |/10)*Entropy(S poor ) = 1.571 - (0.7) * (1.842) - (0.3) * 0 = 1.571 - 1.2894 = 0.2816 This means that the first node in the decision tree will be the weather attribute. As an exercise, convince yourself why this scored (slightly) higher than the parents attribute - remember what entropy means and look at the way information gain is calculated. From the weather node, we draw a branch for the values that weather can take: sunny, windy and rainy: Weekend (Example) Weather Parents Money Decision (Category) W1 Sunny Yes Rich Cinema W2 Sunny No Rich Tennis W3 Windy Yes Rich Cinema W4 Rainy Yes Poor Cinema W5 Rainy No Rich Stay in W6 Rainy Yes Poor Cinema W7 Windy No Poor Cinema W8 Windy No Rich Shopping W9 Windy Yes Rich Cinema W10 Sunny No Rich Tennis Page 1 of 3 Decision Tree Learning 11/7/2011 http://www.doc.ic.ac.uk/~sgc/teaching/v231/lecture11.html
Background image of page 1
Now we look at the first branch. S sunny = {W1, W2, W10}. This is not empty, so we do not put a default categorisation leaf node here. The categorisations of W1, W2 and W10 are Cinema, Tennis and Tennis respectively. As these are not all the same, we cannot put a categorisation leaf node here. Hence we put an attribute node here, which we will leave blank for the time being. Looking at the second branch, S windy = {W3, W7, W8, W9}. Again, this is not empty, and they do not all belong to the same class, so we put an attribute node here, left blank for now. The same situation happens with the third branch, hence our amended tree looks like this: Now we have to fill in the choice of attribute A, which we know cannot be weather, because we've already removed that from the list of attributes to use. So, we need to calculate the values for Gain(S sunny , parents) and Gain(S sunny , money). Firstly, Entropy(S sunny ) = 0.918. Next, we set S to be S sunny = {W1,W2,W10} (and, for this part of the branch, we will ignore all the other examples). In effect, we are interested only in this part of the table: Hence we can calculate: Gain(S sunny , parents) = 0.918 - (|S yes |/|S|)*Entropy(S yes ) - (|S no |/|S|)*Entropy(S no ) = 0.918 - (1/3)*0 - (2/3)*0 = 0.918 Gain(S sunny , money) = 0.918 - (|S rich |/|S|)*Entropy(S rich ) - (|S poor |/|S|)*Entropy(S poor ) = 0.918 - (3/3)*0.918 - (0/3)*0 = 0.918 - 0.918 = 0 Notice that Entropy(S yes ) and Entropy(S no ) were both zero, because S yes contains examples which are all in the same category (cinema), and S no similarly contains examples which are all in the same category (tennis). This should make it more obvious why we use information gain to choose attributes to put in nodes. Given our calculations, attribute A should be taken as parents. The two values from parents are yes and no, and we will draw a branch from the node for each of these. Remembering that we replaced the set S by the set S Sunny , looking at S yes , we see that the only example of this is W1. Hence, the branch for yes stops at a Weekend (Example) Weather Parents Money Decision (Category) W1 Sunny Yes Rich Cinema W2 Sunny No Rich Tennis W10 Sunny No Rich Tennis Page 2 of 3 Decision Tree Learning 11/7/2011 http://www.doc.ic.ac.uk/~sgc/teaching/v231/lecture11.html
Background image of page 2
Show entire document
Sign up to view the entire interaction

Top Answer

Dear Student, I assessed your homework question, but unfortunately was not able to complete... View the full answer

Sign up to view the full answer

Why Join Course Hero?

Course Hero has all the homework and study help you need to succeed! We’ve got course-specific notes, study guides, and practice tests along with expert tutors.

-

Educational Resources
  • -

    Study Documents

    Find the best study resources around, tagged to your specific courses. Share your own to gain free Course Hero access.

    Browse Documents
  • -

    Question & Answers

    Get one-on-one homework help from our expert tutors—available online 24/7. Ask your own questions or browse existing Q&A threads. Satisfaction guaranteed!

    Ask a Question
Ask a homework question - tutors are online