EE_740_HW2 - EE 740 Homework 2 22 January 08 Edmund G....

Info iconThis preview shows pages 1–4. Sign up to view the full content.

View Full Document Right Arrow Icon
EE 740 Homework 2 22 January 08 Edmund G. Zelnio Wright State University —————————————————————————————————————————————————— Problem 2.12, Cover and Thomas Example of Joint Entropy. A fair coin is flipped until the Frst head occurs. Let X denote the number of flips required. The joint PD± is shown in ±igure 1. 1/3 1/3 01 / 3 0 0 1 1 X Y ±igure 1: Joint Probability Density ±unction a. ±ind H ( X ), H ( Y ) H ( X )= X x X p ( x ) log p ( x )( 1 ) H ( X X x ∈{ 0 , 1 } p ( x ) log p ( x ) H ( X 2 3 · log 2 3 1 3 · log 1 3 =0 . 9183 H ( Y 1 3 · log 1 3 2 3 · log 2 3 . 9183 b. ±ind H ( X/Y ), H ( Y/X ) H ( X/Y X x X X y Y p ( x, y ) · log p ( x/y 2 ) H ( X/Y X x ∈{ 0 , 1 } X y ∈{ 0 , 1 } p ( x, y ) · log p ( x/y ) H ( X/Y 1 3 · log 1 1 3 · log 1 2 0 · log 0 1 3 · log 1 2 = 2 3 H ( X x X X y Y p ( x, y ) · log p ( y/x 3 ) H ( 1 3 · log 1 2 1 3 · log 1 2 0 · log 0 1 3 · log 1= 2 3 c. ±ind H ( X, Y ). H ( X, Y X x X X y Y p ( x, y ) · log p ( x, y 4 ) H ( X, Y 1 3 · log 1 3 1 3 · log 1 3 0 · log 0 1 3 · log 1 3 =1 . 5850 d. ±ind H ( Y ) H ( ). ±rom above H ( Y ) H ( . 9183 . 6667 = . 2516. e. ±ind I ( X, Y ). I ( X, Y X x X X y Y p ( x, y ) · log p ( x, y ) / ( p ( x ) · p ( y )) (5)
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
H(X) H(Y) H(Y/X) H(X/Y) I(X,Y) H(X,Y) H(X)-H(X/Y)= I(X,Y) =H(Y)-H(Y/X) Figure 2: Venn Diagram of Entropy Related Quantities f. Draw a Venn Diagram of the above information quantities. Shown in Figure 2. —————————————————————————————————————————————————— Problem 2.13, Cover and Thomas Inequality. Show that ln x 1 1 /x for x> 0. We will show this by plotting the functions (Reference Figure 3) over two large and disparate regions illustrating that log(x) is larger. 10 -16 10 -15 10 -14 10 -13 10 -12 10 -11 10 -10 -10 16 -10 14 -10 12 -10 10 -10 8 -10 6 -10 4 -10 2 -10 0 log(x) 1 - 1/x (a) 10 0 10 2 10 4 10 6 10 8 10 10 10 -5 10 -4 10 -3 10 -2 10 -1 10 0 10 1 10 2 Log(x) 1 - 1/x (b) Figure 3: Logarithmic Plots Showing That Log(x) is larger. —————————————————————————————————————————————————— 2.14 Cover and Thomas Entropy of a Sum. Let X and Y be random variables that take on values x 1 ,x 2 3 , ··· r and y 1 ,y 2 3 , y s , respectively. Let X + Y = Z . a. Show that H ( Z/X )= H ( Y/X ). Hence, p ( Z = z/X = x P ( Y = z x/X = x ). H ( Z/X X x X p ( x ) H ( Z/X = x )( 6 ) X X
Background image of page 2
= X x X p ( x ) X y Y p ( Y = z x/X = x ) · log p ( Y = z x/X = x ) = X x X p ( x ) H ( Y/X = x )= H ( ) Now if X and Y are independent, I ( X, Y X x,y p ( x, y ) log p ( x, y ) p ( x ) · p ( y ) =0 (8) 0= H ( Y ) H ( ) H ( H ( Y ) Now, since H ( Z ) H ( Z/X ), H ( Z/X H ( ), and H ( H ( Y ); we have H ( Z ) H ( Y ). Similarly, with a dual analysis we could establish that H ( Z ) H ( X ). b. Give an example of (necessarily dependent) random variables in which H ( X ) >H ( Z )and H ( Y ) ( Z ).
Background image of page 3

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Image of page 4
This is the end of the preview. Sign up to access the rest of the document.

Page1 / 9

EE_740_HW2 - EE 740 Homework 2 22 January 08 Edmund G....

This preview shows document pages 1 - 4. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online