Coursehero >> New York >> Cornell >> CS 478

Course Hero has millions of student submitted documents similar to the one below including study guides, homework solutions, papers, exam answer keys and textbook solutions.

478 CS Machine Learning: Homework 4 Suggested Solutions 1 The Na Bayes Independence Assumption ve P (X = (0, 1, 1, 1, 1) | Y = +1) P (X1 = 0, X2 = 1, X3 = 1, X4 = 1, X5 = 1 | Y = +1) P (X1 = 0, X2 = 1 | Y = +1) P (X1 = 0 | Y = +1)P (X2 = 1 | Y = +1) 0.2 0.6 0.12 (a) For case (i) = = = = = For case (ii) (1) P (X = (0, 1, 1, 1, 1) | Y = +1) = P (X1 = 0, X2 = 1, X3 = 1, X4 = 1, X5 = 1 | Y = +1) = P (X1 = 0 | Y = +1)P (X2 = 1 | Y = +1)P (X3 = 1 | Y = +1)P (X4 = 1 | Y = +1)P (X5 = 1 | Y = +1) = 0.2 0.64 = 0.02592 (2) (b) Due to the Naive Bayes assumption, the multivaraite Naive Bayes estimate of the probability is the same as the computation of case (ii) in part (a) above, and is 0.02592. So the two cases cannot be distinguished. (c) For case (i), using the true probability, we have P (X = (0, 1, 1, 1, 1), Y = +1) = P (X = (0, 1, 1, 1, 1) | Y = +1)P (Y = +1) = 0.12 0.5 = 0.06 And P (X = (0, 1, 1, 1, 1), Y = 1) = P (X = (0, 1, 1, 1, 1) | Y = 1)P (Y = 1) = 0.8 0.4 0.5 = 0.16 Therefore by Bayes rule the example X = (0, 1, 1, 1, 1) will be classi ed as 1. 1 (3) (4) For case (ii) using the true probability, we have P (X = (0, 1, 1, 1, 1), Y = +1) = P (X = (0, 1, 1, 1, 1) | Y = +1)P (Y = +1) = 0.02592 0.5 = 0.01296 And P (X = (0, 1, 1, 1, 1), Y = 1) = P (X = (0, 1, 1, 1, 1) | Y = 1)P (Y = 1) = 0.8 0.44 0.5 = 0.01024 By Bayes rule the example X = (0, 1, 1, 1, 1) will be classi ed as +1. Now since the Naive Bayes assumption is true in case (ii), the multivariate Naive Bayes algorithm will classify X as +1 for both case (ii) and also case (i) (in which it makes a wrong independence assumption). So the Naive Bayes classi er gives a classi cation that is di erent from that arise from the true probabilities in case (i). (d) For case (i) where X2 , X3 , X4 , X5 are completely dependent, after splitting using of these attributes it becomes pointless to split using any of the remaining three since they give zero information gain (since the attributes always have same values). So the decision tree would split using X1 and only one of X2 , X3 , X4 , X5 and then stop. For case (ii) where X2 , X3 , X4 , X5 are independent, we obtain some positive information gain by splitting on each of these attributes due to independence. Since we assume that the decision tree algorithm does not do early stopping, a full tree will be grown using all 5 attributes. Thus we can see that the decision tree algorithm can distinguish between case (i) and case (ii) by considering the information gain of di erent attributes. Indeed it is not hard to see that the decision trees produced in each of the two cases gives the best decision rule possible, i.e., acheiving the Bayes error. Comment: Most of you have no trouble with this question, except some small careless mistakes while manipulating the probabilities. (5) (6) 2 Na Bayes for Text Classi cation ve (a) The rst assumption is the Naive Bayes assumption that the conditional probability of observing a document given the class is the product of the conditional probability of observing each individual word given the class. The second simplifying assumption is that the conditional probability of a word given the class is independent of the position of the word in the document. 2 (b) The multinomial Naive Bayes model take into account the length of the document in its probabilities, while the multivariate model is only concerned with whether the word appears in the document (so that the representation of a single-word document oil is the same as a ten-word document with the word oil repeated 10 times, with same probability). On the other hand, the multivariate model explicits take into account the probability of the absence of di erent words in its probability model, while the multinomial model does not. (c) crude oil: multivariate: train: 0.9132 test: 0.9139 multinomial: train: 0.9816 test: 0.9738 mac: multivariate: train: 0.8366 test: 0.8092 multinomial: train: 0.9618 test: 0.8827 An example python script for the program is appended at the end of this solutions. (d) Top ten 254 286 955 1678 2937 1282 3144 1734 1426 1372 words from bpd opec bbl herrington pdvsa api saudis aegean texaco re neries the multinomial model: 835.784508254 445.829340754 370.166397806 363.182126149 342.229311179 314.292224552 293.339409582 279.370866269 279.370866269 251.433779642 Top ten words from the multivariate model: 3 955 254 1372 731 1395 1282 1868 2064 1678 1426 bbl bpd re neries distillate postings api petroleos wti herrington texaco 468.0 442.0 358.8 327.6 312.0 296.4 280.8 249.6 249.6 249.6 (e) using default parameters for C and linear kernel crude train: 0.9841 crude test: 0.9755 mac train: 0.8975 mac test: 0.8504 Comment: In part (c) we accept accuracies within 1 to 2 percent as correct, and points are deducted by how far o the accuracies are. In part (d) most of you did the estimation correctly and obtain the right words. In part (e) since the question did not specify what parameters to use for the SVM, using the default parameters or some careful selection of C will both result in full credit. 3 Linear Discriminant Analysis P (Y = +1)e 2 1 (a) According to Bayes rule, we classify an example as +1 if x + 2 > P (Y = 1)e 2 2 1 x 2 log P (Y = +1) 1 x + 2 > log P (Y = 1) 1 x 2 2 1 P (Y = +1) [ x + 2 x 2 ] + log( )>0 2 P (Y = 1) 1 P (Y = +1) [x x 2 + x + + + x x + 2 x ] + log >0 2 P (Y = 1) + 2 2 P (Y = +1) ( + ) x + log >0 2 P (Y = 1) (7) Therefore we can see that the LDA classi cation rule can be rewritten in the form of 2 2 P (Y =+1) sign(w x + b), with w = + and b = + + log P (Y = 1) 2 (b) In this particular case + = 1, = 1, so w = 2. As the positive class is twice as likely as the negative class, we have P (Y = +1)/P (Y = 1) = 2, so b = log 2. Therefore the decision rule is sign(2x + log 2) = sign(x + 1/2 log 2). An positive example becomes an error if it lies to the left of 1/2 log 2, and likewise a negative 4 example becomes an error if it lies to the right of the 1/2 log 2. Therefore the error rate of this decision rule is: 1 2 log 2 P (Y = +1) 1 (x 1)2 e 2 dx + P (Y = 1) 2 1 log 2 2 1 2 1 2 log 2 1 (x+1)2 e 2 dx 2 = = = 2 3 2 3 1 2 1 2 1 log 2 1 (x 1)2 1 e 2 dx + 3 2 1 (x 1)2 1 e 2 dx + 3 2 1 (x+1)2 e 2 dx 2 1 (x 1)2 e 2 dx 2 1 x2 e 2 dx 2 (8) log 2 log 2 1 2 2 2 log 2 1 1 x2 1 e 2 dx + 3 3 2 2 1 = (0.0891) + (0.2567) 3 3 = 0.1450 log 2 1 Therefore the error rate is 0.1450 for the LDA. (c) Suppose a xed b is given. For yi = 1, we have i = 1 b xi for xi < 1 b, and 0 otherwise. For yi = 1, we have i = 1 + b + xi for xi > 1 b, and 0 otherwise. (All these can be checked by looking at the constraints). Therefore the objective can be expressed as: n i i=1 (9) (1 b xi ) + (1 + b + x) 1 i n,yi = 1,xi > 1 b = 1 i n,yi =1,xi <1 Since b the examples xi , yi are generated according to the LDA model, in the large sample limit, we can replace the sum with integral and obtaint the following expression for the objective (actually, the objective divided by the sample size n): 1 b P (Y = 1) 1 (x 1)2 (1 b x) exp( )dx + P (Y = 1) 2 2 1+b (x + 1)2 1 )dx (1 + b + x) exp( 2 2 1 b 2 = 3 = 2 3 1 b b 1 (x 1)2 1 (1 b x) exp( )dx + 2 3 2 2 b 1 (x 1)2 (1 + b x) exp( )dx 2 2 1 y 1 ( b y) exp( )dy + 2 3 2 1 y2 (b y) exp( )dy 2 2 (10) To nd out the minimizer, we can di erentiate the objective with respect to b and set it to zero. For the rst term, using the product rule for di erentiation and di erentiation 5 under the integral sign, the derivative is: d db = 1 y2 ( b y) exp( )dy 2 2 b b y2 b2 b2 1 1 1 exp( )dy b exp( ) ( b) exp( ) 2 2 2 2 2 2 = ( b) (11) where is the cumulative distribution function of the standard normal distribution. Similarly we have derivative of the second term to be (b). Setting the derivative to zero, we have 1 2 ( b) + (b) = 0 3 3 2 ( b) = (b) (12) Since the standard normal distribution is symmetric, we have ( b) = 1 (b), and the above equation reduces to (b) = 2/3. Looking up the normal distribution table we have b = 0.43. Using similar calculations as in part (b), we can nd out the error rate of this decision boundary is 0.1457, which is greater than the LDA decision boundary. Thus the SVM is not Bayes optimal in this case. Comment: Most of you have no problem with part (a) and (b). The work in part (c) is more complex than what this course required and was intended as a challenge type of question. Many of you do come up with good solutions, either through hand calculations or by solving the problem numerically. 4 4.1 Appendix Multinomial Naive Bayes # 24 March 2008 # naive bayes classifier import math import sys SIZE_VOCAB = 27658 category = "crude" in_file = open(category+".train","r") count = {} 6 count[-1]=0 count[1]=0 # initialize the word count array word_count = {} word_count[-1] = map(lambda x:0, range(SIZE_VOCAB+1)) word_count[1] = map(lambda x:0, range(SIZE_VOCAB+1)) class_word_count={} class_word_count[-1]=0 class_word_count[1]=0 for line in in_file.readlines(): vector = line.split() label = int(vector[0]) count[label]+=1 words = vector[1:] for word in words: (feature_num, freq) = word.split(":") feature_num = int(feature_num) freq = int(freq) word_count[label][feature_num]+=freq class_word_count[label]+=freq in_file.close() # conditional probabilities, with Laplacian smoothing # binary-valued cond_prob = {} cond_prob[-1] = map(lambda x:0, range(SIZE_VOCAB+1)) cond_prob[1] = map(lambda x:0, range(SIZE_VOCAB+1)) for feature_num in range(1,SIZE_VOCAB+1): cond_prob[-1][feature_num] = (word_count[-1][feature_num]+1.0)\ /(class_word_count[-1]+SIZE_VOCAB) cond_prob[1][feature_num] = (word_count[1][feature_num]+1.0)\ /(class_word_count[1]+SIZE_VOCAB) # training set accuracy in_file = open(category+".train","r") num_error = 0 7 num_examples = 0 for line in in_file.readlines()[1:]: vector = line.split() label = int(vector[0]) words = vector[1:] pos_log_lr = math.log(count[1]) neg_log_lr = math.log(count[-1]) for word in words: feature_num = int(word.split(":")[0]) pos_log_lr += math.log(cond_prob[1][feature_num]) neg_log_lr += math.log(cond_prob[-1][feature_num]) if pos_log_lr-neg_log_lr>0: prediction = 1 else: prediction = -1 if prediction!=label: num_error+=1 # to avoid division by zero # exponeitate the log-likelihood ratio if prediction==1: # compute P(X,Y=1)/(P(X,Y=1)+P(X,Y=-1)) by # 1/(1+exp(log(P(X,Y=-1)/P(X,Y=+1)))) print "incorrectly predict +1" print 1/(1+math.exp(neg_log_lr-pos_log_lr)) else: print "incorrectly predict -1" print 1/(1+math.exp(pos_log_lr-neg_log_lr)) num_examples+=1 print print print print "Total number of errors:" num_error "Accuracy:" float(num_examples-num_error)/num_examples in_file.close() # find top 10 most predictive words ratio_list = zip(cond_prob[1],cond_prob[-1]) ratio_list = map(lambda x: x[0]/x[1], ratio_list[1:]) rankings = zip(ratio_list, range(1,SIZE_VOCAB+1)) rankings.sort() 8 # load list of words in_file = open("words.train.top","r") vocab_list = in_file.readlines() in_file.close() vocab_list = map(lambda x: x.strip(), vocab_list) topten = rankings[-20:] topten.reverse() for pair in topten: out_str = str(pair[1])+" "+vocab_list[pair[1]-1]+" "+str(pair[0]) print out_str 4.2 Multivariate Naive Bayes # 26 March 2008 # naive bayes classifier import math SIZE_VOCAB = 27658 category = "crude" in_file = open(category+".train","r") count = {} count[-1]=0 count[1]=0 # initialize the word count array word_count = {} word_count[-1] = map(lambda x:0, range(SIZE_VOCAB+1)) word_count[1] = map(lambda x:0, range(SIZE_VOCAB+1)) for line in in_file.readlines(): vector = line.split() label = int(vector[0]) count[label]+=1 words = vector[1:] for word in words: feature_num = int(word.split(":")[0]) word_count[label][feature_num]+=1 in_file.close() 9 # conditional probabilities, with Laplacian smoothing # binary-valued cond_prob = {} cond_prob[-1] = {} cond_prob[1] = {} cond_prob[-1][0] = map(lambda x:0, range(SIZE_VOCAB+1)) cond_prob[-1][1] = map(lambda x:0, range(SIZE_VOCAB+1)) cond_prob[1][0] = map(lambda x:0, range(SIZE_VOCAB+1)) cond_prob[1][1] = map(lambda x:0, range(SIZE_VOCAB+1)) for feature_num in range(1,SIZE_VOCAB+1): cond_prob[-1][1][feature_num] = (word_count[-1][feature_num]+1.0)/(count[-1]+2) cond_prob[-1][0][feature_num] = ((count[-1]-word_count[-1][feature_num])+1.0)/(count[-1]+2) cond_prob[1][1][feature_num] = (word_count[1][feature_num]+1.0)/(count[1]+2) cond_prob[1][0][feature_num] = ((count[1]-word_count[1][feature_num])+1.0)/(count[1]+2) # training set accuracy in_file = open(category+".test","r") num_error = 0 num_examples = 0 print count[1] print count[-1] base_lr = math.log(float(count[1])/count[-1]) for i in range(1, SIZE_VOCAB+1): base_lr += (math.log(cond_prob[1][0][i]) - math.log(cond_prob[-1][0][i])) for line in in_file.readlines(): vector = line.split() label = int(vector[0]) words = vector[1:] log_lr = base_lr for word in words: feature_num = int(word.split(":")[0]) log_lr += (math.log(cond_prob[1][1][feature_num])-math.log(cond_prob[-1][1][feature_num]) - math.log(cond_prob[1][0][feature_num]) + math.log(cond_prob[-1][0][feature_num])) if log_lr>0: prediction = 1 else: prediction = -1 10 if prediction!=label: num_error+=1 print num_error num_examples+=1 in_file.close() print float(num_examples-num_error)/num_examples # find top 10 most predictive words ratio_list = zip(cond_prob[1][1],cond_prob[-1][1]) ratio_list = map(lambda x: x[0]/x[1], ratio_list[1:]) rankings = zip(ratio_list, range(1,SIZE_VOCAB+1)) rankings.sort() # load list of words in_file = open("words.train.top","r") vocab_list = in_file.readlines() in_file.close() vocab_list = map(lambda x: x.strip(), vocab_list) topten = rankings[-20:] topten.reverse() for pair in topten: out_str = str(pair[1])+" "+vocab_list[pair[1]-1]+" "+str(pair[0]) print out_str 11

Find millions of documents here - Study Guides, Homework Solutions, Papers, Exam Answer Keys and more. Course Hero has millions of course related materials that will enable you to learn better, faster and get an A in all your courses.
Below is a small sample set of documents: