cond_mutual_info_2 - I X,A Y,Z = X x,a,y,z p x,a,y,z log p...

Info iconThis preview shows pages 1–2. Sign up to view the full content.

View Full Document Right Arrow Icon
Conditional Mutual Information Ohad Shamir December 2, 2008 We have seen in class the standard definition of mutual information I ( X ; Y ) between two random variables: I ( X ; Y ) = X x,y p ( x,y ) log p ( x,y ) p ( x ) p ( y ) . We have also seen that mutual information satisfies the identities I ( X ; Y ) = H ( X ) - H ( X | Y ) = H ( Y ) - H ( Y | X ) . (1) Conditional mutual information measures the mutual information between two random variables, conditioned on an additional random variable: I ( X ; Y | Z ) = X z p ( z ) I ( X ; Y | Z = z ) = X x,y,z p ( z ) p ( x,y | z ) log p ( x,y | z ) p ( x | z ) p ( y | z ) . It satisfies the identities I ( X ; Y | Z ) = H ( X | Z ) - H ( X | Y,Z ) = H ( Y | Z ) - H ( Y | X,Z ) . Notice the similarity between this and Eq. (1). Finally, notice that instead of talking about mutual information between single random variables, we can define mutual information between sets of ran- dom variables: we can think of each set just as one big random variable. For instance, the mutual information between X,A on one hand and Y,Z on the other hand is
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Background image of page 2
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: I ( X,A ; Y,Z ) = X x,a,y,z p ( x,a,y,z ) log p ( x,a,y,z ) p ( x,a ) p ( y,z ) . Similar to the identities above, I ( X,A ; Y,Z ) = H ( X,A )-H ( X,A | Y,Z ) = H ( Y,Z )-H ( Y,Z | X,A ) . Notice that the ; operator tells us between which sets of random variables the mutual information is taken. Similarly, we can also use conditional mutual in-formation, when the conditioning is over a set of random variables. For instance, I ( X,A ; Y,Z | B,C ) = H ( X,A | B,C )-H ( X,A | Y,Z,B,C ) = H ( Y,Z | B,C )-H ( Y,Z | X,A,B,C ) . 1 Finally, we have the chain rule for mutual information (not hard to prove), which states that for any random variables X 1 ,...,X n ,Y , I ( X 1 ,X 2 ,...,X n ; Y ) = n X i =1 I ( X i ; Y | X 1 ,X 2 ,...,X i-1 ) . 2...
View Full Document

{[ snackBarMessage ]}

Page1 / 2

cond_mutual_info_2 - I X,A Y,Z = X x,a,y,z p x,a,y,z log p...

This preview shows document pages 1 - 2. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online