HW2s[1]

# HW2s[1] - ECE 534 Elements of Information Theory Fall 2010...

This preview shows pages 1–3. Sign up to view the full content.

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: ECE 534: Elements of Information Theory, Fall 2010 Homework 2 Solutions Ex. 2.28 (Kenneth S. Palacio Baus) Mixing increases entropy. Show that the entropy of the probability distribution ( p 1 ,... ,p i ,... ,p j ,... ,p m ) is less than the entropy of the distribution ( p 1 ,... , p i + p j 2 ,... , p i + p j 2 ,...,p m ). Show that in general any transfer of probability that makes the distribution more uniform increases the entropy. Solution We need to compare the entropies for each distribution. There are only two probabilities which are different between the two distributions. Those probabilities determine the distribution with higher entropy. Lets define the probability distribution p ( x ) = ( p 1 ,...,p i ,...,p j ,...,p m ) for variable X, and the prob- ability distribution p ( y ) = ( p 1 ,..., p i + p j 2 ,..., p i + p j 2 ,...,p m ) for variable Y. Then, we need to compare the entropies H ( X ) vs. H ( Y ). H ( X ) = − p 1 log 2 p 1 + ... − p i log 2 p i + ... − p j log 2 p j + ... − p m log 2 p m H ( Y ) = − p 1 log 2 p 1 + ... − p i + p j 2 log 2 p i + p j 2 + ... − p i + p j 2 log 2 p i + p j 2 + ... − p m log 2 p m Selecting the terms that are different in the two previous expressions we obtain: For H ( X ): − p i log 2 p i − p j log 2 p j For H ( Y ): − p i + p j 2 log 2 p i + p j 2 − p i + p j 2 log 2 p i + p j 2 Applying the Log Sum inequality, theorem 2.7.1 of [ ? ], on the expression for H ( X ): 1 ( p i log 2 p i + p j log 2 p j ) ≥ ( p i + p j )log 2 p i + p j 2 Which implies: − ( p i log 2 p i + p j log 2 p j ) ≤ − ( p i + p j )log 2 p i + p j 2 Hence, we can conclude that the entropy of distribution for X is less than the one for Y. In general, for making a distribution more uniform, we should include more similar terms. Lets write the distribution of X as: p ( x ) = ( p 1 ,...,p i ,...,p j ,...,p k ,...,p m ), and the probability distribu- tion p ( y ) = ( p 1 ,..., p i + p j + p k 3 ,..., p i + p j + p k 3 ,..., p i + p j + p...
View Full Document

## This note was uploaded on 01/19/2012 for the course ECE 534 taught by Professor Natashadevroye during the Fall '10 term at Ill. Chicago.

### Page1 / 6

HW2s[1] - ECE 534 Elements of Information Theory Fall 2010...

This preview shows document pages 1 - 3. Sign up to view the full document.

View Full Document
Ask a homework question - tutors are online