cond_mutual_info_2

# cond_mutual_info_2 - I X,A Y,Z = X x,a,y,z p x,a,y,z log p...

This preview shows pages 1–2. Sign up to view the full content.

Conditional Mutual Information Ohad Shamir December 2, 2008 We have seen in class the standard deﬁnition of mutual information I ( X ; Y ) between two random variables: I ( X ; Y ) = X x,y p ( x,y ) log p ( x,y ) p ( x ) p ( y ) . We have also seen that mutual information satisﬁes the identities I ( X ; Y ) = H ( X ) - H ( X | Y ) = H ( Y ) - H ( Y | X ) . (1) Conditional mutual information measures the mutual information between two random variables, conditioned on an additional random variable: I ( X ; Y | Z ) = X z p ( z ) I ( X ; Y | Z = z ) = X x,y,z p ( z ) p ( x,y | z ) log p ( x,y | z ) p ( x | z ) p ( y | z ) . It satisﬁes the identities I ( X ; Y | Z ) = H ( X | Z ) - H ( X | Y,Z ) = H ( Y | Z ) - H ( Y | X,Z ) . Notice the similarity between this and Eq. (1). Finally, notice that instead of talking about mutual information between single random variables, we can deﬁne mutual information between sets of ran- dom variables: we can think of each set just as one big random variable. For instance, the mutual information between X,A on one hand and Y,Z on the other hand is

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: I ( X,A ; Y,Z ) = X x,a,y,z p ( x,a,y,z ) log p ( x,a,y,z ) p ( x,a ) p ( y,z ) . Similar to the identities above, I ( X,A ; Y,Z ) = H ( X,A )-H ( X,A | Y,Z ) = H ( Y,Z )-H ( Y,Z | X,A ) . Notice that the ; operator tells us between which sets of random variables the mutual information is taken. Similarly, we can also use conditional mutual in-formation, when the conditioning is over a set of random variables. For instance, I ( X,A ; Y,Z | B,C ) = H ( X,A | B,C )-H ( X,A | Y,Z,B,C ) = H ( Y,Z | B,C )-H ( Y,Z | X,A,B,C ) . 1 Finally, we have the chain rule for mutual information (not hard to prove), which states that for any random variables X 1 ,...,X n ,Y , I ( X 1 ,X 2 ,...,X n ; Y ) = n X i =1 I ( X i ; Y | X 1 ,X 2 ,...,X i-1 ) . 2...
View Full Document

{[ snackBarMessage ]}

### Page1 / 2

cond_mutual_info_2 - I X,A Y,Z = X x,a,y,z p x,a,y,z log p...

This preview shows document pages 1 - 2. Sign up to view the full document.

View Full Document
Ask a homework question - tutors are online