midterm_soln - University of Toronto Department of...

Info iconThis preview shows pages 1–2. Sign up to view the full content.

View Full Document Right Arrow Icon

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: University of Toronto Department of Electrical February 28, 2006 & Computer Engineering ECE1502S — Information Theory Midterm Test Solution 1 . ( Keeping Both Eyes Open ) We know that X and Y are independent and X and Z are independent. Does this mean that X is independent of the pair Y,Z ? The answer is no! For example, let Y and Z be chosen independently and uniformly at random from { , 1 } , and let X = Y ⊕ Z where ⊕ denotes modulo-two addition. Then X is independent of Y and X is independent of Z , but I ( X ; Y,Z ) = 1 bit. 2 . ( Entropy Inequalities ) In general, we have H ( X 2 | X 1 ) ≤ 1 2 H ( X 1 ,X 2 ) ≤ H ( X 1 ). To see that 2 H ( X 2 | X 1 ) ≤ H ( X 1 ,X 2 ) observe that H ( X 1 ,X 2 ) = H ( X 1 ) + H ( X 2 | X 1 ) (chain rule) = H ( X 2 ) + H ( X 2 | X 1 ) (by stationarity) ≥ H ( X 2 | X 1 ) + H ( X 2 | X 1 ) (conditioning does not increase entropy) = 2 H ( X 2 | X 1 ) Similarly, H ( X 1 ,X 2 ) = H ( X 1 ) + H ( X 2 | X 1 ) (chain rule) ≤ H ( X 1 ) + H ( X 2 ) (conditioning does not increase entropy) = H ( X 1 ) + H ( X 1 ) (by stationarity) = 2 H ( X 1 ) . (More cleverly, we may observe that H ( X 1 ,X 2 ) = H ( X 1 ) + H ( X 2 | X 1 ), where clearly H ( X 2 | X 1 ) ≤ H ( X 2 ) = H ( X 1 ). Thus the numbers H ( X 2 | X 1 ), 1 2 H ( X 1 ,X 2 ) and H ( X 1 ) form a non-decreasing arithmetic sequence, and the result follows.)form a non-decreasing arithmetic sequence, and the result follows....
View Full Document

This note was uploaded on 10/24/2011 for the course ELECTRICAL ECE 571 taught by Professor Kelly during the Spring '11 term at University of Illinois, Urbana Champaign.

Page1 / 3

midterm_soln - University of Toronto Department of...

This preview shows document pages 1 - 2. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online