HW2s[1] - ECE 534: Elements of Information Theory, Fall...

Info iconThis preview shows pages 1–3. Sign up to view the full content.

View Full Document Right Arrow Icon

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: ECE 534: Elements of Information Theory, Fall 2010 Homework 2 Solutions Ex. 2.28 (Kenneth S. Palacio Baus) Mixing increases entropy. Show that the entropy of the probability distribution ( p 1 ,... ,p i ,... ,p j ,... ,p m ) is less than the entropy of the distribution ( p 1 ,... , p i + p j 2 ,... , p i + p j 2 ,...,p m ). Show that in general any transfer of probability that makes the distribution more uniform increases the entropy. Solution We need to compare the entropies for each distribution. There are only two probabilities which are different between the two distributions. Those probabilities determine the distribution with higher entropy. Lets define the probability distribution p ( x ) = ( p 1 ,...,p i ,...,p j ,...,p m ) for variable X, and the prob- ability distribution p ( y ) = ( p 1 ,..., p i + p j 2 ,..., p i + p j 2 ,...,p m ) for variable Y. Then, we need to compare the entropies H ( X ) vs. H ( Y ). H ( X ) = p 1 log 2 p 1 + ... p i log 2 p i + ... p j log 2 p j + ... p m log 2 p m H ( Y ) = p 1 log 2 p 1 + ... p i + p j 2 log 2 p i + p j 2 + ... p i + p j 2 log 2 p i + p j 2 + ... p m log 2 p m Selecting the terms that are different in the two previous expressions we obtain: For H ( X ): p i log 2 p i p j log 2 p j For H ( Y ): p i + p j 2 log 2 p i + p j 2 p i + p j 2 log 2 p i + p j 2 Applying the Log Sum inequality, theorem 2.7.1 of [ ? ], on the expression for H ( X ): 1 ( p i log 2 p i + p j log 2 p j ) ( p i + p j )log 2 p i + p j 2 Which implies: ( p i log 2 p i + p j log 2 p j ) ( p i + p j )log 2 p i + p j 2 Hence, we can conclude that the entropy of distribution for X is less than the one for Y. In general, for making a distribution more uniform, we should include more similar terms. Lets write the distribution of X as: p ( x ) = ( p 1 ,...,p i ,...,p j ,...,p k ,...,p m ), and the probability distribu- tion p ( y ) = ( p 1 ,..., p i + p j + p k 3 ,..., p i + p j + p k 3 ,..., p i + p j + p...
View Full Document

Page1 / 6

HW2s[1] - ECE 534: Elements of Information Theory, Fall...

This preview shows document pages 1 - 3. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online