CryptoInfo

# CryptoInfo - Handout#12 ma187s Cryptography The Entropy...

This preview shows pages 1–4. Sign up to view the full content.

Handout #12 ma187s: Cryptography April 25, 2006 The Entropy Function: Identities and Inequalities Theorem 1 H ( X, Y )= H ( X )+ H ( Y | X H ( Y H ( X | Y ) Proof. We prove only the ﬁrst equality since the second is proved in an entirely analogous manner. Recall that the deﬁnition of H ( X, Y ) may be written as H ( X, Y X a X b P [ X = a, Y = b ] log 2 1 P [ X = a, Y = b ] (1) Since P [ X = a, Y = b ]= P [ X = a ] × P [ X = a, Y = b ] P [ X = a ] = P [ X = a ] × P [ Y = b | X = a ] we may rewrite (1) as H ( X, Y X a X b P [ X = a, Y = b ] log 2 1 P [ X = a ] × P [ Y = b | X = a ] = X a X b P [ X = a, Y = b ] log 2 1 P [ X = a ] + X a X b P [ X = a, Y = b ] log 2 1 P [ Y = b | X = a ] = X a P [ X = a ] log 2 1 P [ X = a ] + X a X b P [ X = a ] × P [ Y = b | X = a ] log 2 1 P [ Y = b | X = a ] = X a P [ X = a ] log 2 1 P [ X = a ] + X a P [ X = a ] X b P [ Y = b | X = a ] log 2 1 P [ Y = b | X = a ] = H ( X H ( Y | X ) QED Theorem 2 For any two random variables X and Y we always have H ( X | Y ) H ( X ) (2) and equality holds if and only if X and Y are independent. Proof. From our deﬁnitions we get H ( X | Y X b P [ Y = b ] X a P [ X = a | Y = b ] log 2 1 P [ X = a | Y = b ] = X b P [ Y = b ] X a P [ X = a, Y = b ] P [ Y = b ] log 2 1 P [ X = a | Y = b ] = X b X a P [ X = a, Y = b ] log 2 1 P [ X = a | Y = b ] = X a P [ X = a ] X b P [ Y = b | X = a ] log 2 1 P [ X = a | Y = b ] (3) 1

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
Handout #12 ma187s: Cryptography April 25, 2006 Since for a given a , the conditional probabilities P [ Y = b | X = a ] add up to 1, we can use the convex function inequality X b m b log 2 x b log 2 X b m b x b ! (4) with m b = P [ Y = b | X = a ] and x b = 1 P [ X = a | Y = b ] = P [ Y = b ] P [ X = a, Y = b ] and obtain that for any a we have X b P [ Y = b | X = a ] log 2 1 P [ X = a | Y = b ] log 2 X b P [ Y = b | X = a ] 1 P [ X = a | Y = b ] ! (5) = log 2 X b P [ X = a, Y = b ] P [ X = a ] × P [ Y = b ] P [ X = a, Y = b ] ! = log 2 X b P [ Y = b ] P [ X = a ] ! (6) = log 2 1 P [ X = a ] Using this inequality in (3) we ﬁnally derive that H [ X | Y ] X a P [ X = a ] log 2 1 P [ X = a ] = H ( X ) . (7) Which is what we wanted to prove. Now recall that the inequality in (4) can turn into an equality only if all the x b are the same. So we can have an equality in (5) only if for that particular a all the probabilities P [ X = a | Y = b ] are the same as b varies. In other words there must be a constant c such that we have for all b P [ X = a, Y = b ] P [ Y = b ] = P [ X = a | Y = b ]= c, or better P [ X = a, Y = b c × P [ Y = b ] . (8) Summing over b this gives us the equality P [ X = a c substituting this back into (8) gives P [ X = a, Y = b P [ X = a ] × P [ Y = b ] . Now since this must hold for all a and b for (7) to be an equality, we deduce that equality can hold true if and only if X and Y are independent. This result has an important corollary: 2
Handout #12 ma187s: Cryptography April 25, 2006 Theorem 3 For any two random variables X and Y we have H ( X, Y ) H ( X )+ H ( Y ) with equality holding if and only if X and Y are independent Proof. Combining the equality given by Theorem 1 with the inequality of Theorem 2 we get H ( X, Y )= H ( X H ( Y | X ) H ( X H ( Y ) , as desired. Since we have used Theorem 2 we see that equality can only hold true if X and Y are independent. QED Theorem 4 For a random variable X which takes only k values we always have H ( X ) log 2 k with equality if and only if X takes all its values with equal probability Proof. The deﬁnition gives H ( X X b VALUES P [ X = b ] log 2 1 P [ X = b ] Using again the inequality in (4), this time with m b = P [ X = b ] and x b = 1 P [ X = b ] gives H ( X ) log 2 ± X b VALUES P [ X = b ] 1 P [ X = b ] ² = log 2 ± X b

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}

### Page1 / 8

CryptoInfo - Handout#12 ma187s Cryptography The Entropy...

This preview shows document pages 1 - 4. Sign up to view the full document.

View Full Document
Ask a homework question - tutors are online