7.35 - EE 376A\/Stat 376A Prof T Weissman Information Theory Friday Solutions to Practice Final Problems These problems are sampled from a couple of the

# 7.35 - EE 376A/Stat 376A Prof T Weissman Information Theory...

This preview shows page 1 - 4 out of 13 pages.

EE 376A/Stat 376A Information Theory Prof. T. Weissman Friday, March 17, 2006 Solutions to Practice Final Problems These problems are sampled from a couple of the actual finals in previous years. 1. ( 20 points) Errors and erasures. Consider a binary symmetric channel (BSC) with crossover probability p . - - · · · · · · · ·3 Q Q Q Q Q Q Q Qs 1 0 1 0 1 - p 1 - p p p A helpful genie who knows the locations of all bit flips offers to convert flipped bits into erasures. In other words, the genie can transform the BSC into a binary erasure channel. Would you use his power? Be specific. Solution: Errors and erasures. Although it is very tempting to accept the genie’s offer, on a second thought, one realizes that it is disadvantageous to convert the bit flips into erasures when p is large. For example, when p = 1 , the original BSC is noiseless, while the “helpful” genie will erase every single bit coming out from the channel. The capacity C 1 ( p ) of the binary symmetric channel with crossover probability p is 1 - H ( p ) while the capacity C 2 ( p ) of the binary erasure channel with erasure probability p is 1 - p . One would convert the BSC into a BEC only if C 1 ( p ) C 2 ( p ), that is, p p * = . 7729 . (See Figure 1.) 2. (20 points) Code constraint. What is the capacity of a BSC( p ) under the constraint that each of the codewords has a proportion of 1’s less than or equal to α , i.e., 1 n n X i =1 X i ( w ) α, for w ∈ { 1 , 2 , . . . , 2 nR } . (Pay attention when α > 1 / 2.) 1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 p 1-H(p) 1-p pstar Solution: Code constraint. Using the similar argument for the capacity of Gaussian channels under the power constraint P , we find that the capacity C of a BSC( p ) under the proportion constraint α is C = max p ( x ): EX α I ( X ; Y ) . Now under the Bernoulli( π ) input distribution with π α , we have I ( X ; Y ) = H ( Y ) - H ( Y | X ) = H ( Y ) - H ( Z | X ) = H ( Y ) - H ( Z ) = H ( π * p ) - H ( p ) , (1) where π * p = (1 - π ) p + π (1 - p ). (Breaking I ( X ; Y ) = H ( X ) - H ( X | Y ) = H ( X ) - H ( Z | Y ) is way more complicated since Z and Y are correlated.) Now when α > 1 / 2 , we have max π H ( π * p ) - H ( p ) = 1 - H ( p ) , with the capacity-achieving π * = 1 / 2 . On the other hand, when α 1 / 2 , π * = α achieves the maximum of (1); hence C = H ( α * p ) - H ( p ) . 3. ( 20 points) Partition. Let ( X, Y ) denote height and weight. Let [ Y ] be Y rounded off to the nearest pound. (a) Which is greater I ( X ; Y ) or I ( X ; [ Y ]) ? (b) Why? 2
Solution: Partition. (a) I ( X ; Y ) I ( X ; [ Y ]) . (b) Data processing inequality. 4. ( 20 points) Amplify and forward. We cascade two Gaussian channels by feeding the (scaled) output of the first channel into the second. · ¶‡ - - ? - ? · ¶‡ q - - · ¶‡ - ? X 1 P X 2 P Y 1 Y 2 Z 1 N (0 , N ) α Z 2 N (0 , N ) Thus noises Z 1 and Z 2 are independent and identically distributed according to N (0 , N ), EX 2 1 = EX 2 2 = P, Y 1 = X 1 + Z 1 , Y 2 = X 2 + Z 2 , and X 2 = αY 1 , where the scaling factor α is chosen to satisfy the power constraint EX 2 2 = P .

#### You've reached the end of your free preview.

Want to read all 13 pages?

• Spring '10
• sd
• Information Theory, rate distortion, Zi Xi Yi, rate distortion function