Section CS221 7
1
CS 221 Section 7: Bayesian Networks
1. Chocolate Bayesian Networks Suppose you have nally graduated and have your dream job of supervising the labeling chocolate that is being produced in a factory. There are two levers controlling production, one that controls whether the chocolate is plain or almond, and another that controls whether there is coconut or not. The levers are faulty, and will randomly switch to the other position occasionally. The levers stay in the position theyre in with probability .7 and switch to the other setting .3, with each of the two lever switching independently. At the start of the day, the lever is in either setting with equal probability. You are working at the nal conveyor belt and you cannot see what the levers are set to from your vantage point. You can only tell what color the chocolate is from seeing it on the converyor belt. Fortunately you do have the following table of probabilities for the color given whats inside. Inside Plain (N) Almond (A) Coconut (C) Almond + Coconut (B) P(Color=Light (L) |Inside) 0.1 0.3 0.8 0.9 P(Color=Dark (D) |Inside) 0.9 0.7 0.2 0.1
You would like to know the probability that the chocolate is a certain avor based on what you observe its color to be. In this problem we will show how a Bayesian Network can be used to help you in your task. We will let Xi represent the state of the two levers at time step i, and Yi represent the observation at time step i. So, the domain of Xi is {Plain (N), Almond (A), Coconut, Almond + Coconut (B)} and the domain of Yi is {Light (L), Dark (D)}. (a) Draw a Bayes Net corresponding to this situation Answer: The avor of the chocolate at time i (Yi ) depends only upon the position of the levers at time i (Xi ). The position of the levers at time i depends upon the position of the levers at time i 1, if i > 1. If i = 1, then each position is equally likely. We can capture this in the following network
CS221 Section 7
2
(b) Suppose the rst chocolate that you observe is Dark (Y1 = D). What is the probability that this chocolate is coconut avored (i.e. what is P (X1 = C |Y1 = D))? Answer: We begin by using Bayes Rule to rewrite what we want in terms of what we have. P (X1 = C |Y1 = D) = P (Y1 = D|X1 = C )P (X1 = C ) P (Y1 = D)
The terms in the numerator can be read o of our chart, but the term on the denominator takes a little more work.
P (Y1 = D) =
x1 X1
P (X1 = x1 , Y1 = D) P (X1 = x1 )P (Y1 = D|X1 = x1 )
x1 X1
= = 0.25
P (Y1 = D|X1 = x1 )
x1 X1
= 0.25 (0.9 + 0.7 + 0.2 + 0.1) = 0.25 (1.9)
Thus, the answer we want is 0.2 0.25 P (Y1 = D|X1 = C )P (X1 = C ) = = 0.2/1.9 = 0.105 P (Y1 = D) 1.9 0.25
CS221 Section 7
3
(c) Suppose that you now observe that the next chocolate is light (Y2 = L). What is the probability that this chocolate is coconut avored? Calculate P (X2 = C |Y1 = D, Y2 = L)? Answer: =D X1 ) First, we need P (X1 |Y1 = D) = P (Y1 P (Y|1 =DP (X1 ) . Note that P (X1 ) = .25 for all X1 . ) X1 P A C A+C So, P (Y1 = D|X1 )P (X1 ) .225 .175 .05 .025 P (X1 |D) .474 .368 .105 .053
P (X2 = C |Y1 = D, Y2 = L) P (X2 = C, Y1 = D, Y2 = L) = P (Y1 = D, Y2 = L) P (Y1 = D)P (X2 = C |Y1 = D)P (Y2 = L|X2 = C, Y1 = D) = P (Y1 = D)P (Y2 = L|Y1 = D) P (Y2 = L|X2 = C )P (X2 = C |Y1 = D) = P (Y2 = L|Y1 = D) P (Y2 = L|X2 = C ) xX1 P (X2 = C |X1 = x)P (X1 = x|Y1 = D) = P (Y2 = L|Y1 = D) The numerator P (Y2 = L|X2 = C ) xX1 P (X2 = C |X1 = x)P (X1 = x|Y1 = D) is equal to .8 (.21 .474 + .09 .368 + .49 .105 + .21 .053) = .156. To normalize, we need to do the same calculation for the other possible values of X2 , which are .034, .090, and .132. So, P (X2 = C |Y1 = D, Y2 = L) = .156/.412 = .379. 2. Bayes Net Inference: Asia network Consider the Asia network considered in lecture:
Visit to Asia
Smoking
Tuberculosis
Lung Cancer Bronchitis
Abnormality in Chest
X-ray
Dyspnea
CS221 Section 7
4
(a) Perform variable elimination for this network, in order to get the probability distribution P (D). You can pick any order to eliminate variables. Answer: We have that P (D) =
A,B,L,T,S,X,V
P (V )P (T | V )P (S )P (L | S )P (B | S )P (A | L, T )P (X | A)P (D | A, B )
where we eliminate according to the order in the summation. Our computation proceeds from the inside out. We rst eliminate V . Thus, we rst compute, for each pair of values t and v , the product P (t | v )P (v ), and then sum out to obtain f1 (T ) = P (T | V )P (V )
V
For example, if T (Tuberculosis) has the values {no, mild, severe}, this process would end up computing three numbers, one for every value of T . The next phase would compute, f2 (A) = X P (X | A), which is a function that has a value for each value of A. The third phase would compute, f3 (B, L) = S P (B | S )P (L | S )P (S ), which is a factor that contains a value for every pair of values b and l for the variables B and L. Thus, if Bronchitis has three values and lung cancer has four, then this would be a total of 12 values. The fourth phase is to sum out T . For that, we need to multiply everything that mentions T . We need to multiply P (A | T, L) with f1 (T ): f4 (A, L) =
T
P (A | T, L)f1 (T )
The fth phase is to sum out L: f5 (A, B ) =
L
f3 (B, L) f4 (A, L)
The nal phase is to sum out A and B . This would involve multiplying everything thats left, and summing out A and B . Thus, we would have P (D | A, B )f5 (A, B )f2 (A)
A,B
The result is a factor over D, which gives us the distribution over the possible values of D. (b) Now we introduce evidence into the picture. Assume we observe s1 (the person smokes) and x0 (the X-ray came out negative). Perform variable elimination with evidence, in order to get P (D, s1 , x0 ). Show how to get P (D | s1 , x0 ) from that. Answer: First, we reduce this problem to computing P (D, s1 , x0 ), i.e., P (d, s1 , x0 ) for every value d of D. We can use that as follows: P (d | s1 , x0 ) = P (d, s1 , x0 ) = P (s1 , x0 ) P (d, s1 , x0 1 ) 0 d P (d, s , x )
In other words, once we compute a factor over D that represents the joint probability P (D, s1 , x0 ), we simply renormalize it to get the desired conditional probability.
CS221 Section 7
5
How do we compute P (D, s1 , x0 )? Let us go back to the joint distribution. To compute any individual value P (d, s1 , x0 ), we want to sum out only those entries in the joint where the values D, S, X take on these particular values. In other words: P (d, s1 , x0 ) =
A,B,L,T,V
P (V )P (T | V )P (s1 )P (L | s1 ) P (B | s1 )P (A | L, T )P (x0 | A)P (d | A, B )
We can execute this computation in exactly the same way, by eliminating the unnecessary variables all except D, S, X in the usual way. We do not eliminate D because we are interested in getting a factor over its values; we do not eliminate S and X because their values are xed. Let us use the same elimination ordering that we used in our rst computation before (the one that generated the f factors rather than the g factors): i. ii. iii. iv. v. Eliminating V : This phase is unchanged; it generates the same f1 (T ). Eliminating X : This phase now involves no operations. Eliminating S : This phase also involves no operations. Eliminating T : This phase is unchanged; it generates the same f4 (A, L). Eliminating L: This phase is now dierent. Note that, before, the summation over S in phase (3) caused us to sum out various factors into a single factor f3 (B, L). In this computation, this did not happen. In our case, the summation looks like: f5 (A) =
L
f4 (A, L)P (L | s1 )
Note that this term doesnt depend on S , because only one value of S is involved; i.e., P (L | s1 ) has exactly one number for every value of L. Also note that f5 , unlike f5 , does not depend on B . Intuitively, observing S decorrelates B and L, rendering them conditionally independent. From the algorithms perspective, we never compute f3 (B, L), which forced us to introduce the dependence on B . vi. Eliminating B : f6 (A) = P (B | s1 )P (D | A, B )
B
vii. Eliminating A: f7 ( D ) =
A 1 0
f6 (A)P (x0 | A)f5 (A)
viii. The result: To get P (D, s , x ), the result needs to be multiplied by the remaining term P (s1 ). (Note that, for getting the conditional P (D | s1 , x0 ), this last multiplication by a constant factor is irrelevant.) (c) What can you say about the values in the factor generated by summing out X-ray? Answer: The factor for X-ray is f (X, A). Its equal to P (X | A). Summing out X-ray gives X P (X | A) = 1. So the value of the factor f (A) generated by summing out X is always 1 for all values of A. (d) Suppose all the variables are binary, except for A, which has a domain of size 3. For each node in the network, write down the number of independent parameters in the CPD, and the total. Answer: At each node n, the number of independent parameters = (N umV alues(n) 1) n P a(n) N umV alues(n )
CS221 Section 7
6
V = 1, T = 2 (2 1) = 2, S = 1, L = 2, B = 2, A = 2 2 (3 1) = 8, X = 3 (2 1) = 3, D = 3 2 (2 1) = 6. Total = 25. 3. Learning using Naive Bayes Classiers One very common technique for classication is the Naive Bayes classier. Most simply, assume that our feature space consists of the binary-valued features X1 , . . . , Xn , and that we want to classify each instance into two classes, d0 or d1 . Use x0 to denote the false i value (0) of Xi and x1 to denote its true value (1). The Naive Bayes classier uses a i Bayesian network of the following structure to do the classication:
D X1 X2 X3 ... Xk
It learns the parameters for the network from the training set, and then classies a new instance into the most likely class. (a) Write down a formula for deciding whether a new instance x1 , . . . , xn belongs to d1 . Your formula should be written directly in terms of the CPD entries in the network: P (dl ) (l = 0, 1), and P (xij | dl ) (i = 1, . . . , n, l, j = 0, 1). Answer: We need P (d1 | x1 , . . . , xn ) >1 P (d0 | x1 , . . . , xn ) If we apply Bayes rule, P (d1 | x1 , . . . , xn ) = = And likewise P (d0 | x1 , . . . , xn ) = P (x1 | d0 )P (x2 | d0 ) . . . P (xn | d0 )P (d0 ) P (x1 , . . . , xn ) P (x1 , . . . , xn | d1 )P (d1 ) P (x1 , . . . , xn ) P (x1 | d1 )P (x2 | d1 ) . . . P (xn | d1 )P (d1 ) P (x1 , . . . , xn )
When we take the ratio of the above two probabilities, the denominators cancel: P (x1 | d1 )P (x2 | d1 ) . . . P (xn | d1 )P (d1 ) P (d1 | x1 , . . . , xn ) = P (d0 | x1 , . . . , xn ) P (x1 | d0 )P (x2 | d0 ) . . . P (xn | d0 )P (d0 ) Therefore, we prefer class d1 to d0 if P (x1 | d1 )P (x2 | d1 ) . . . P (xn | d1 )P (d1 ) >1 P (x1 | d0 )P (x2 | d0 ) . . . P (xn | d0 )P (d0 ) (b) Show that your classier from (a) is equivalent to a linear separator between the d0 instances and the d1 instances. In other words, show that we can construct weights
CS221 Section 7
7
w0 , w1 , . . . , wn using the CPT entries such that an instance x is classied as d0 if n hw (x) = w0 + i=1 wi xi < 0, and as d1 if hw (x) > 0 . You can ignore ties where both d0 and d1 are equally likely. (Hint: Consider a transformation to your formula from (a) that would result in a summation.) Answer: First we transform the equation above by taking the log of both sides: log( P (x1 | d1 )P (x2 | d1 ) . . . P (xn | d1 )P (d1 ) > log(1) P (x1 | d0 )P (x2 | d0 ) . . . P (xn | d0 )P (d0 ) log(P (xi | d1 )) log(P (xi | d0 )) > 0
i
log(P (d1 )) log(P (d0 )) + Then a good choice for weights is: w0 wi
= log(P (d1 )) log(P (d0 )) +
i
log(P (x0 | d1 )) log(P (x0 | d0 )) i i
= log(P (x1 | d1 )) log(P (x1 | d0 )) (log(P (x0 | d1 )) log(P (x0 | d0 ))) i i i i
where i = 1, . . . , n. Intuitively, the idea is that when xi = 0, the dierence log(P (x0 | i d1 )) log(P (x0 )) must be introduced into the equation, so the only way to do that is to i place it in w0 . But when xi = 1, the above dierence must be replaced by log(P (x1 | i d1 )) log(P (x1 )). i