solns00 - University of Toronto Department of Electrical...

Info iconThis preview shows pages 1–3. Sign up to view the full content.

View Full Document Right Arrow Icon

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: University of Toronto Department of Electrical November 8, 2000 & Computer Engineering ECE1502S — Information Theory Midterm Test Solutions 1. ( Matching Distributions ) (a) Call a particular ordering of Q optimal if D ( P || Q ) is minimized. Suppose an optimal ordering exists in which i < j , but q i > q j . Let Q be the distribution obtained by swapping the i th and j th probability masses. Then D ( P || Q )- D ( P || Q ) = p i log p i q i + p j log p j q j- p i log p i q j- p j log p j q i = p i log q j + p j log q i- p i log q i- p j log q j = ( p i- p j ) | {z } ≤ (log q j- log q i ) | {z } < ≥ , with equality if and only if p i = p j . We see that, in general, swapping the i th and j th probability masses reduces the relative entropy, so Q can be optimal in this situation only if p i = p j . But if p i = p j then swapping q i and q j does not affect the relative entropy. Thus sorting the probabilities yields an optimal ordering. (b) Consider now D ( Q || P ) and assume p 1 > 0. Again suppose that an optimal ordering exists in which i < j , but q i > q j . Let Q be the distribution obtained by swapping the i th and j th probability masses. Then D ( Q || P )- D ( Q || P ) = q i log q i p i + q j log q j p j- q j log q j p i- q i log q i p j = q i log p j- q i log p i- q j log p j + q j log p i = ( q i- q j ) | {z } > (log p j- log p i ) | {z } ≥ = ≥ with equality if and only if p i = p j . By the same argument as above, sorting the probabilities yields an optimal ordering. (c) If Q has one mass equal to zero, then by the result of (b), we can set q 1 = 0. We wish to select q 2 ,q 3 ,...,q m so that D ( Q || P ) is minimized. Setting up the Lagrangian L ( q 2 ,...,q m ,λ ) = m X i =1 q i ln( q i /p i ) + λ ( m X i =1 q i- 1) , differentiating with respect to q i ( i > 1) and setting the result to zero, we find that ln( q i /p i ) + 1 + λ = 0 , i.e., q i /p i is a constant, independent of i . The constant is chosen to make ∑ m i =2 q i = 1. We find that q i = ( if i = 0; p i 1- p 1 if 2 ≤ i ≤ m 1 2. ( Huffman Coding with Costs ) The Huffman procedure minimizes ∑ p i l i , for a set of “weights” { p i } that sum to unity. To show that this procedure works for any arbitrary set of weights, simply divide by the sum of the weights....
View Full Document

This note was uploaded on 10/24/2011 for the course ELECTRICAL ECE 571 taught by Professor Kelly during the Spring '11 term at University of Illinois, Urbana Champaign.

Page1 / 6

solns00 - University of Toronto Department of Electrical...

This preview shows document pages 1 - 3. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online