This preview shows pages 1–2. Sign up to view the full content.
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
Unformatted text preview: University of Toronto Department of Electrical February 28, 2006 & Computer Engineering ECE1502S — Information Theory Midterm Test Solution 1 . ( Keeping Both Eyes Open ) We know that X and Y are independent and X and Z are independent. Does this mean that X is independent of the pair Y,Z ? The answer is no! For example, let Y and Z be chosen independently and uniformly at random from { , 1 } , and let X = Y ⊕ Z where ⊕ denotes modulotwo addition. Then X is independent of Y and X is independent of Z , but I ( X ; Y,Z ) = 1 bit. 2 . ( Entropy Inequalities ) In general, we have H ( X 2  X 1 ) ≤ 1 2 H ( X 1 ,X 2 ) ≤ H ( X 1 ). To see that 2 H ( X 2  X 1 ) ≤ H ( X 1 ,X 2 ) observe that H ( X 1 ,X 2 ) = H ( X 1 ) + H ( X 2  X 1 ) (chain rule) = H ( X 2 ) + H ( X 2  X 1 ) (by stationarity) ≥ H ( X 2  X 1 ) + H ( X 2  X 1 ) (conditioning does not increase entropy) = 2 H ( X 2  X 1 ) Similarly, H ( X 1 ,X 2 ) = H ( X 1 ) + H ( X 2  X 1 ) (chain rule) ≤ H ( X 1 ) + H ( X 2 ) (conditioning does not increase entropy) = H ( X 1 ) + H ( X 1 ) (by stationarity) = 2 H ( X 1 ) . (More cleverly, we may observe that H ( X 1 ,X 2 ) = H ( X 1 ) + H ( X 2  X 1 ), where clearly H ( X 2  X 1 ) ≤ H ( X 2 ) = H ( X 1 ). Thus the numbers H ( X 2  X 1 ), 1 2 H ( X 1 ,X 2 ) and H ( X 1 ) form a nondecreasing arithmetic sequence, and the result follows.)form a nondecreasing arithmetic sequence, and the result follows....
View
Full
Document
This note was uploaded on 10/24/2011 for the course ELECTRICAL ECE 571 taught by Professor Kelly during the Spring '11 term at University of Illinois, Urbana Champaign.
 Spring '11
 Kelly

Click to edit the document details