This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
Unformatted text preview: ECE 1502 — Information Theory Problem Set 5 solutions 1 April 11, 2006 9.1 Differential Entropy. (a) Exponential distribution. h ( f ) = Z ∞ λe λx [ln λ λx ] dx (1) = ln λ + 1 nats. (2) = log e λ bits. (3) (b) Laplace density. h ( f ) = Z ∞∞ 1 2 λe λ  x  [ln 1 2 + ln λ λ  x  ] dx (4) = ln 1 2 ln λ + 1 (5) = ln 2 e λ nats. (6) = log 2 e λ bits. (7) (c) Sum of two normal distributions. The sum of two normal random variables is also normal, so applying the result derived the class for the normal distribution, since X 1 + X 2 ∼ N ( μ 1 + μ 2 ,σ 2 1 + σ 2 2 ), h ( f ) = 1 2 log 2 πe ( σ 2 1 + σ 2 2 ) bits. (8) 9.3 Mutual information for correlated normals. X Y ∼ N 2 , σ 2 ρσ 2 ρσ 2 σ 2 (9) Using the expression for the entropy of a multivariate normal derived in class h ( X,Y ) = 1 2 log(2 πe ) 2  K  = 1 2 log(2 πe ) 2 σ 4 (1 ρ 2 ) . (10) Since X and Y are individually normal with variance σ 2 , h ( X ) = h ( Y ) = 1 2 log 2 πeσ 2 . (11) Hence I ( X ; Y ) = h ( X ) + h ( Y ) h ( X,Y ) = 1 2 log(1 ρ 2 ) . (12) 1 Solutions to problems from the text are supplied courtesy of Joy A. Thomas. 1 (a) ρ = 1. In this case, X = Y , and knowing X implies perfect knowledge about Y . Hence the mutual information is infinite, which agrees with the formula. (b) ρ = 0. In this case, X and Y are independent, and hence I ( X ; Y ) = 0, which agrees with the formula. (c) ρ = 1. In this case, X = Y , and again the mutual information is infinite as in the case when ρ = 1. 9.6 Scaling. Let Y = A X . Then the density of Y is g ( y ) = 1  A  f ( A 1 y ) . (13) Hence h ( A X ) = Z g ( y ) ln g ( y ) d y (14) = Z 1  A  f ( A 1 y ) ln f ( A 1 y ) log  A  d y (15) = Z 1  A  f ( x ) [ln f ( x ) log  A  ]  A  d x (16) = h ( X ) + log  A  . (17) 10.3 The two look Gaussian channel.The two look Gaussian channel....
View
Full Document
 Spring '10
 323
 Normal Distribution, Probability theory, 1 m, Joy A. Thomas, 0 1 m

Click to edit the document details