ECE 563, Fall 2011
Homework I
Issued: August 30th, 2011
Due: September 13th, 2011
1. Entropy and Majorization. Entropy after Mixing Symbols.
The interpretation of entropy as a measure of uncertainty suggests that more
uniform distributions have larger ent
1
ECE 534: Elements of Information Theory
Midterm Exam I (Spring 2007) Problem 1 [20 pts] What are the relations (, =, ) between the following pairs of expressions? Explain why. 1) H (5X ) and H (X ); 2) H (X0 |X1 ) and H (X0 |X1 , X1 ); 3) H (X1 , X2 , .
1
ECE 534: Elements of Information Theory Solutions to Midterm Exam (Spring 2006)
Problem 1 [20 pts.] A discrete memoryless source has an alphabet of three letters, xi , i = 1, 2, 3, with probabilities 0.4, 0.4, and 0.2, respectively. (a) Find the binary
ECE 534 Information Theory - Midterm 2
Nov.4, 2009. 3:30-4:45 in LH103.
You will be given the full class time: 75 minutes. Use it wisely! Many of the problems have short answers; try to nd shortcuts. You may bring and use two 8.5x11 double-sided crib she
University of Illinois at Chicago Department of Electrical and Computer Engineering
ECE 534: Information Theory
Fall 2009 Midterm 1 - Solutions
NAME:
This exam has 4 questions, each of which is worth 15 points. You will be given the full class time: 75 m
ECE 534: Elements of Information Theory, Fall 2011
Homework 1
Solutions
Problem 2.2 (Hamid Dadkhahi) Let Y = g (X ). Suppose PX (x) and PY (y ) denote the probability mass functions of random variables X and Y , respectively. If g is an injective mapping,
University of Toronto November 8, 2000
Department of Electrical & Computer Engineering
ECE1502S Information Theory
Midterm Test Solutions
1. (Matching Distributions) (a) Call a particular ordering of Q optimal if D(P |Q) is minimized. Suppose an optimal o
University of Toronto F. R. Kschischang ECE1502F - Information Theory
Department of Electrical & Computer Engineering
Final Examination Solutions
December 13, 2000 1. Short Snappers (a) I(X; X) = H(X), the entropy of X. (b) False. Although I(X; Y ) = H(Y
University of Toronto November 9, 2007
Department of Electrical & Computer Engineering
ECE1502F - Information Theory
Midterm Test Solution
1. (Noisy Gates) (a) The noisy XOR gate induces a binary symmetric channel (BSC) with crossover probability p. As sh
University of Toronto February 28, 2006
Department of Electrical & Computer Engineering
ECE1502S - Information Theory
Midterm Test Solution
1. (Keeping Both Eyes Open) We know that X and Y are independent and X and Z are independent. Does this mean that X
ECE1502F Information Theory
Final Examination Solutions
December 11, 2007 1. (a) Let X and Y be independent Bernoulli(1/2) random variables and let Z = X Y , where denotes modulo-two addition. Since X and Y are independent, we have I(X; Y ) = 0. Since X i
University of Toronto F. R. Kschischang
ECE 1502S Information Theory
Department of Electrical & Computer Engineering
Solutions for Final Examination of April 17, 2006 1. Short Snappers (a) False. For example, C = cfw_0, 0 satisfies the Kraft inequality, b
University of Illinois
Fall 2011
ECE 563: Homework 6
Issued: October 16th, 2011; Due: October 27th, 2011
1. Let C1 be a code with codelengths li = log2 (i) and let C2 be a code with codelengths
li = log2 (i2 ), where i cfw_2, 3, . . ..
Can C1 and C2 be pr
inequalities.
(1/3; 1/3; 1/4; 1/12).
H (X, Y |Z ) H ( code.
Construct a Humann X |Z ), Show that there are two dierent optimal codes with codeword lengths (1; 2; 3, 3)
and (2; 2; 2; 2). Z ) I (X ; that there exists an optimal code such that some of its c
Homework Set #3
Homework Set #3
Homework Set #3
1. Random walk in a cube.
1. Random walk in a cube.
A bird ies from room to room in a 3 3 3 cube (equally likely through each interior
1. Random walk in a cube.
A bird ies from room to room in a 3 3 3 cube (
ECE 563, Fall 2011
Homework 3 Solution
Question 1:
3.13, Cover and Thomas
(n)
(n)
Calculation of Typical Set To clarify the notion of typical set A , and the smallest set of high probability B ,
we will calculate the set for a simple example. consider a s
the minimum value for H (X, Y, Z )?
4. Bottleneck.
Suppose a (non-stationary) Markov chain starts in one of n states, necks down to k < n
states, and then fans back to m > k states. Thus X1 X2 X3 , X1 cfw_1, 2, . . . , n,
2. The value .of ,a , X cfw_1, 2,
Question 1:
ECE 563, Fall 2011
Homework 2 Solution
Solution: Random questions.
We assume that X, Q1 , Q2 are jointly independent. We also assume A(X, Q) is a
deterministic function; i.e., the answer is a function of the question Q and the object
X.
(a)
(a
ECE 563, Fall 2011
Homework I Solution
Parts of the solutions are copied from the solution manual of Elements of Information Theory by
Cover and Thomas.
Question 1: Entropy and Majorization. Entropy after Mixing Symbols.
Check class notes for a complete s
University of Illinois at Urbana-Champaign
ECE 563: Information Theory
Fall 2008
Midterm 2
Wednesday, November 19, 2008
Name:
This is a closed-book exam. You may consult both sides of two sheets of notes, typed in
font size 10 or equivalent handwriting s
EE 478
Multiple User Information Theory
Handout #22
November 13, 2008
Homework Set #6 Solutions
1. Solution:
(a) Codebook generation: For each m0 [1 : 2nR0 ], m01 [1 : 2nR01 ], and m02 [1 : 2nR02 ]
generate un (m0 , m01 , m02 ) n pU0 (u0i ). Choose R11 >
EE 478
Multiple User Information Theory
Handout #18
Thursday, October 30, 2008
Homework Set #6
Due: Tuesday, November 11, 2008.
1. Marton inner bound with common messages. Consider the Marton inner bound with common
messages described on page 5-37 of the
EE 478
Multiple User Information Theory
Handout #20
November 6, 2008
Homework Set #5 Solutions
1. Solution:
(a) From the lecture notes, the capacity region CBC (g1 , g2 , P ) of the Gaussian broadcast
channel is given by the set of rate pairs (R1 , R2 ) s
EE 478
Multiple User Information Theory
Handout #15
Thursday, October 23, 2008
Homework Set #5
Due: Thursday, October 30, 2008.
1. Duality between Gaussian broadcast and multiple access channels.
Consider the following Gaussian broadcast and multiple acce
EE 478
Multiple User Information Theory
Handout #17
October 28, 2008
Homework Set #4 Solutions
1. Solution: We need to bound the probability of error events described in the lecture notes.
Note that
n
n
n
1
Pcfw_(n , U2 , X1 (n ), X2 (U2 ), Y n ) T (n) fo
EE 478
Multiple User Information Theory
Handout #12
Thursday, October 16, 2008
Homework Set #4
Due: Thursday, October 23, 2008.
1. Sending correlated sources over a MAC. Consider the joint source-channel coding theorem in
Lecture Notes #4. Complete the pr