7.25 - EE 376A Prof T Weissman Information Theory Friday Homework Set#6(Due 5pm Friday Feb 26 2010 1 Channel capacity with cost constraint There are

# 7.25 - EE 376A Prof T Weissman Information Theory Friday...

This preview shows page 1 - 3 out of 8 pages.

EE 376A Information Theory Prof. T. Weissman Friday, Feb 19, 2010 Homework Set #6 (Due: 5pm Friday, Feb. 26, 2010) 1. Channel capacity with cost constraint There are applications in which some chan- nel input symbols are more costly than others. Letting Λ : |X | 7→ R denote the “cost function”. The per symbol cost of the transmitted sequence X n is Λ( X n ) = 1 n n i =1 Λ( X i ). The expected per symbol cost of the code is then given by EΛ( X n ) = 2 - nR 2 n R X k =1 Λ( x n ( k )) , where x n ( k ) is the k th codeword in the code book. Definition: A rate R is achievable at cost Γ, if > 0 there exists n and a channel code of block-length n, rate R - , E Λ( X n ) Γ+ and probability of error P ( n ) e . Definition: The capacity-cost function of the channel is defined as C (Γ) = sup { R : R is achievable at costΓ } . In words, C (Γ) is the maximum rate of reliable communication when restricting the transmission to an expected cost that does not exceed Γ per channel use. (a) Prove that for a given P Y | X ( y | x ), I ( X ; Y ) is a concave function of P X ( x ). Hint: First show that if f : R n → R , is a convex function, then for any n × m real-valued matrix A , g ( X ) , f ( AX ), g : R m → R , is also a convex function, i.e. linear transformation preserves convexity. (b) Consider the following expression C ( I ) (Γ) = max EΛ( X ) Γ I ( X ; Y ) where the maximum on the right hand side is over all random variables X at the channel input that satisfy the indicated cost constraint. Prove that C ( I ) (Γ) is a non-decreasing concave function in Γ. You may use the result from Part (a). (c) The converse: C (Γ) C ( I ) (Γ). For any sequence of schemes of rate R , EΛ( X n ) Γ, and vanishing probability of error: 1 nR = H ( M ) = I ( M ; Y n ) + H ( M | Y n ) ( a ) I ( M ; Y n ) + n n ( b ) I ( X n ; Y n ) + n n ( c ) H ( Y n ) - n X i =1 H ( Y i | X i ) + n n ( d ) n X i =1 I ( X i ; Y i ) + n n ( e ) n X i =1 C ( I ) (EΛ( X i )) + n n ( f ) nC ( I ) (EΛ( X n )) + n n ( g ) nC ( I ) (Γ) + n n , where n 0 as n → ∞ . Provide explanations for inequalities (a) – (g). Solutions: (a) First we show that if f : R n → R , is a convex function, then for any n × m real-valued matrix A , g ( X ) , f ( AX ), g : R m → R , is also a convex function. Let 0 λ 1, g ( λX 1 + (1 - λ ) X 2 ) = f ( λAX 1 + (1 - λ ) AX 2 ) , λf ( AX 1 ) + (1 - λ ) f ( AX 2 ) , = λg ( X 1 ) + (1 - λ ) g ( X 2 ) . Consequently, linear transformation preserves convexity, equivalently concavity, of functions. Now note that I ( X ; Y ) = H ( Y ) - H ( Y | X ) = H ( Y ) - X x ∈X H ( Y | X = x ) P ( X = x ) . (1) But since P ( Y | X ) is fixed, H ( Y | X = x ) does not depend on P ( X ). Therefore, H ( Y | X  #### You've reached the end of your free preview.

Want to read all 8 pages?

• Spring '10
• sd
• Derivative, γ, Convex function, Concave function
• • •  