{[ promptMessage ]}

Bookmark it

{[ promptMessage ]}

S-72_2410_solution_1

# S-72_2410_solution_1 - S-72.2410 Information Theory Haanp...

This preview shows pages 1–3. Sign up to view the full content.

S-72.2410 Information Theory Haanp¨a¨a & Linja-aho Homework 1, solutions 2009 Homework 1, solutions Deadline: November 9th, 16:00 The box for returning exercises is in the E-wing, 2nd floor corridor. 1. Let p ( x,y ) be given by X downslope Y 0 1 0 1 3 1 2 1 1 6 0 Find: (a) H ( X ) ,H ( Y ) . (b) H ( X | Y ) ,H ( Y | X ) . (c) H ( X,Y ) . (d) H ( Y ) - H ( Y | X ) . (e) I ( X ; Y ) (f) Draw a Venn diagram for the quantities in parts (a) through (e). Solution: (a) H ( X ) = - x ∈X p ( x ) log p ( x ) = - p (0) log( p (0)) - p (1) log( p (1)) = - 5 6 log 5 6 - 1 6 log 1 6 0 . 65 H ( Y ) = - y ∈Y p ( y ) log p ( y ) = - p (0) log( p (0)) - p (1) log( p (1)) 1 (b) H ( X | Y ) = y ∈Y p ( y ) H ( X | Y = y ) = 1 2 H ( X | Y = 0) + 1 2 H ( X | Y = 1) = 1 2 ( - 1 3 log 1 3 - 2 3 log 2 3 ) + 1 2 · 0 0 . 459 H ( Y | X ) = x ∈X p ( x ) H ( Y | X = x ) = 5 6 H ( Y | X = 0) + 1 6 H ( Y | X = 1) = 5 6 ( - 2 5 log 2 5 - 3 5 log 3 5 ) + 1 6 · 0 0 . 809 (c) H ( X,Y ) = - x ∈X y ∈Y p ( x,y ) log p ( x,y ) = - 1 3 log 1 3 - 1 6 log 1 6 - 1 2 log 1 2 - 0 · log 0 1 . 459 (d) H ( Y ) - H ( Y | X ) = 1 - 0 . 809 = 0 . 191 (e) I ( X ; Y ) = H ( X,Y ) - H ( X | Y ) - H ( Y | X ) = 0 . 191 (or H ( Y ) - H ( Y | X ) or H ( X ) - H ( X | Y )) (f) See figure below: Page 1 of 4

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
S-72.2410 Information Theory Haanp¨a¨a & Linja-aho Homework 1, solutions 2009 H ( Y | X ) H ( X | Y ) H ( X ) H ( Y ) I ( X ; Y ) H ( X,Y ) 2. A fair coin is flipped until the first head occurs. Let X denote the number of flips required. (a) Find the entropy H ( X ) in bits. The following expressions may be useful summationdisplay n =0 r n = 1 1 - r summationdisplay n =0 nr n = r (1 - r ) 2 (b) A random variable X is drawn according to this distribution. Find an “efficient” sequence of yes-no questions of the form, “ Is X contained in the set S?
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}

### Page1 / 4

S-72_2410_solution_1 - S-72.2410 Information Theory Haanp...

This preview shows document pages 1 - 3. Sign up to view the full document.

View Full Document
Ask a homework question - tutors are online