{[ promptMessage ]}

Bookmark it

{[ promptMessage ]}

# midtermsol - University of Toronto November 9 2007...

This preview shows pages 1–3. Sign up to view the full content.

University of Toronto Department of Electrical November 9, 2007 & Computer Engineering ECE1502F — Information Theory Midterm Test Solution 1 . ( Noisy Gates ) ( a ) The noisy XOR gate induces a binary symmetric channel (BSC) with crossover probability p . As shown in class and in the text, the capacity of the BSC is C = 1 - H ( p ); this capacity is achieved (for every p ) with a uniform distribution. ( b ) The noisy AND gate with Z Bern (1 / 2) induces a Z-channel: whenever the channel input X = 0, the channel output Y = 0; and whenever the input X = 1, the output Y is equally likely to be 0 or 1. Let P [ X = 1] = p . We have P [ Y = 1] = p/ 2, so H ( Y ) = H ( p/ 2). We may write I ( X ; Y ) = H ( Y ) - H ( Y | X ) = H ( Y ) - H ( Y | X = 0) 0 (1 - p ) - H ( Y | X = 1) 1 p = H ( p/ 2) - p. Recall that d dx H ( x ) = - 1 ln 2 d dx ( x ln x +(1 - x ) ln(1 - x )) = - 1 ln 2 (ln x +1 - ln(1 - x ) - 1) = log 2 1 - x x . Taking the derivative with respect to p , and setting the result to zero, we find 1 2 log((2 - p ) /p ) = 1 or (2 - p ) /p = 4 or p = 2 / 5 . Thus the capacity of the Z-channel is given by H (1 / 5) - 2 / 5 0 . 322, achieved by setting P [ X = 1] = p = 2 / 5. ( c ) The noise OR gate with Z Bern (1 / 2) also induces a Z-channel: whenever the channel input X = 1, the channel output Y = 1; and whenever the channel input X = 0, the output Y is equally likely to be 0 or 1. This is the same channel as in the previous question, except with zeros and ones exchanged. Thus the capacity is the same, namely H (1 / 5) - 2 / 5, achieved by setting P [ X = 1] = 3 / 5. 2 . ( Cover’s Cutting Contraption ) ( a ) Every sequence of cuts induces a binary tree. Each internal node of the tree corresponds to a cut and the children of each node correspond to the two pieces that result from that cut. Associated with each internal node is the cost of cutting 1

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
that node, and this cost is precisely the sum of the lengths of the children of that node. For example, the following figure shows the tree that results from applying the following cuts: i. cut the 1m rod into lengths (0.6,0.4); cost = 1; ii. cut the 0.6m rod into two lengths (0.3,0.3); cost = 0.6; iii. cut the 0.4m rod into lengths (0.2,0.2); cost = 0.4; iv. cut a 0.2m rod into lengths (0.1,0.1); cost = 0.2. 0.1 0.1 0.2 0.3 0.3 0.6 0.2 0.4 1.0 Our objective is to find a tree whose m leaf nodes have values p 1 , p 2 , . . . , p m , and for which the sum of the values over all internal nodes is minimal. Let i denote the depth in the tree of leaf node i having value p i . Let V denote the set of internal nodes, and let C ( v ) denote the value associated with an internal node v . We make use of the following observation: v V C ( v ) = m i =1 p i i . (1) To see this, observe that for each node v V , C ( v ) is given as the sum of the
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}