CryptoInfo

CryptoInfo - Handout #12 ma187s: Cryptography April 25,...

Info iconThis preview shows pages 1–4. Sign up to view the full content.

View Full Document Right Arrow Icon
Handout #12 ma187s: Cryptography April 25, 2006 The Entropy Function: Identities and Inequalities Theorem 1 H ( X, Y )= H ( X )+ H ( Y | X H ( Y H ( X | Y ) Proof. We prove only the first equality since the second is proved in an entirely analogous manner. Recall that the definition of H ( X, Y ) may be written as H ( X, Y X a X b P [ X = a, Y = b ] log 2 1 P [ X = a, Y = b ] (1) Since P [ X = a, Y = b ]= P [ X = a ] × P [ X = a, Y = b ] P [ X = a ] = P [ X = a ] × P [ Y = b | X = a ] we may rewrite (1) as H ( X, Y X a X b P [ X = a, Y = b ] log 2 1 P [ X = a ] × P [ Y = b | X = a ] = X a X b P [ X = a, Y = b ] log 2 1 P [ X = a ] + X a X b P [ X = a, Y = b ] log 2 1 P [ Y = b | X = a ] = X a P [ X = a ] log 2 1 P [ X = a ] + X a X b P [ X = a ] × P [ Y = b | X = a ] log 2 1 P [ Y = b | X = a ] = X a P [ X = a ] log 2 1 P [ X = a ] + X a P [ X = a ] X b P [ Y = b | X = a ] log 2 1 P [ Y = b | X = a ] = H ( X H ( Y | X ) QED Theorem 2 For any two random variables X and Y we always have H ( X | Y ) H ( X ) (2) and equality holds if and only if X and Y are independent. Proof. From our definitions we get H ( X | Y X b P [ Y = b ] X a P [ X = a | Y = b ] log 2 1 P [ X = a | Y = b ] = X b P [ Y = b ] X a P [ X = a, Y = b ] P [ Y = b ] log 2 1 P [ X = a | Y = b ] = X b X a P [ X = a, Y = b ] log 2 1 P [ X = a | Y = b ] = X a P [ X = a ] X b P [ Y = b | X = a ] log 2 1 P [ X = a | Y = b ] (3) 1
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Handout #12 ma187s: Cryptography April 25, 2006 Since for a given a , the conditional probabilities P [ Y = b | X = a ] add up to 1, we can use the convex function inequality X b m b log 2 x b log 2 X b m b x b ! (4) with m b = P [ Y = b | X = a ] and x b = 1 P [ X = a | Y = b ] = P [ Y = b ] P [ X = a, Y = b ] and obtain that for any a we have X b P [ Y = b | X = a ] log 2 1 P [ X = a | Y = b ] log 2 X b P [ Y = b | X = a ] 1 P [ X = a | Y = b ] ! (5) = log 2 X b P [ X = a, Y = b ] P [ X = a ] × P [ Y = b ] P [ X = a, Y = b ] ! = log 2 X b P [ Y = b ] P [ X = a ] ! (6) = log 2 1 P [ X = a ] Using this inequality in (3) we finally derive that H [ X | Y ] X a P [ X = a ] log 2 1 P [ X = a ] = H ( X ) . (7) Which is what we wanted to prove. Now recall that the inequality in (4) can turn into an equality only if all the x b are the same. So we can have an equality in (5) only if for that particular a all the probabilities P [ X = a | Y = b ] are the same as b varies. In other words there must be a constant c such that we have for all b P [ X = a, Y = b ] P [ Y = b ] = P [ X = a | Y = b ]= c, or better P [ X = a, Y = b c × P [ Y = b ] . (8) Summing over b this gives us the equality P [ X = a c substituting this back into (8) gives P [ X = a, Y = b P [ X = a ] × P [ Y = b ] . Now since this must hold for all a and b for (7) to be an equality, we deduce that equality can hold true if and only if X and Y are independent. This result has an important corollary: 2
Background image of page 2
Handout #12 ma187s: Cryptography April 25, 2006 Theorem 3 For any two random variables X and Y we have H ( X, Y ) H ( X )+ H ( Y ) with equality holding if and only if X and Y are independent Proof. Combining the equality given by Theorem 1 with the inequality of Theorem 2 we get H ( X, Y )= H ( X H ( Y | X ) H ( X H ( Y ) , as desired. Since we have used Theorem 2 we see that equality can only hold true if X and Y are independent. QED Theorem 4 For a random variable X which takes only k values we always have H ( X ) log 2 k with equality if and only if X takes all its values with equal probability Proof. The definition gives H ( X X b VALUES P [ X = b ] log 2 1 P [ X = b ] Using again the inequality in (4), this time with m b = P [ X = b ] and x b = 1 P [ X = b ] gives H ( X ) log 2 ± X b VALUES P [ X = b ] 1 P [ X = b ] ² = log 2 ± X b
Background image of page 3

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Image of page 4
This is the end of the preview. Sign up to access the rest of the document.

Page1 / 8

CryptoInfo - Handout #12 ma187s: Cryptography April 25,...

This preview shows document pages 1 - 4. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online