This preview shows page 1. Sign up to view the full content.
Unformatted text preview: inequalities. (1/3; 1/3; 1/4; 1/12). • H (X, Y Z ) ≥ H ( code.
Construct a Huﬀmann X Z ), Show that there are two diﬀerent optimal codes with codeword lengths (1; 2; 3, 3)
and (2; 2; 2; 2). Z ) ≥ I (X ; that there exists an optimal code such that some of its codewords are longer than
• I ((X, Y ); Conclude Z ),
those of the corresponding ShannonFano code.
• H (X, Y, Z ) − H (X, Y ) ≤ H (X, Z ) − H (X ).
Problem 17 A sequence of six symbols are independently drawn according to the distribution of the random
ECE 563
ProblemX . The sequence is encoded symbolbysymbol using an optimal (Huﬀmann) code. The resulted
an arbitrary random variable taking its values from the set
variable 23 (Card shuffling) Let X be IUC, Fall 2011
U
{binary . string is 10110000101. We permutationthe the numbers 1, 2,has. ,ﬁve i.e., T (1), T (2)we. .only know a
1, 2, . . , 52}. Let T be a random know that of source alphabet . . 52, elements, but , . , T (52) is
HW #4
random reordering ofis one of {1, 2, . . .distributions {0, 4; 0, 3; 0T2; 0,independent of{0, 3;Show 0, 2; 0, 2; 0, 05}.
that the distribution the set the two , 52}. We assume that , is 05; 0, 05} and X . 0, 25; that
Determine the distribution of X . Issued: October 4th, 2011
H (T (X )) ≥ 11th,
Due: October H (X ). 2011 Problem 18 We are asked to determine an object by asking yesno questions. The object is drawn randomly
−
Problem 24 set according 1 , X2 , . . . bedistribution. Playing optimally, we need 38.5 questions on the average6 .
from a ﬁnite Let X = Xto a certain a binary memoryless stationary source with P{X1 = 1} = 10
Determine aobject. At least how many elements does theexpected codeword length is smaller than 1/10.
Problemthe variablelength code of X whose per letter ﬁnite set have?
to ﬁnd 1: Problem 25 (Run length coding) Let Huffmann be binary random variables. Let R = (R1 ,binary.)
Problem 19 (Shortest codeword of X1 , . . . , Xn codes) Suppose that we have an optimal R2 , . .
denote coderun lengths of the symbols. in X,1 , . . . , Xn . > p >is,. for example, thethat lengths of the sequence
preﬁx the for the distribution (p1 , . . , pn ) where p1 That . . , pn > 0. Show run
2
1110010001111 is R = (3, 2, 1, 3, 4). Determine the relation between H (X1 , . . . , Xn ), H (R) and H (R, Xn )?
• If p1 > 2/5 then the corresponding codeword has length 1.
Problem 26 (Entropy of a Markov chain) Let X = X1 , X2 , . . . be a binary stationary Markov chain
Problem p < 1/3 then the corresponding codeword has length at least 2.
• If 2:
with state1transition probabilities
Problem 20 (Basketball playoffs) The playoﬀs of NBA are played between team A and team B in1a+ p
1−p
P{X2 = 0X1 = 0} = p, P{X2 = 1X1 = 0} = 1−p, P{X2 = 0X1 = 1} =
, P{X2 = 1X1 = 1} =
.
2
sevengame series that terminates as soon as one of the teams wins four games. Let the random variable X 2
represent P{X1 = 0} What is the games (possible values
Determinethe outcome .of the series of entropy of the source?of X are AAAA or BABABAB or AAABBBB ).
Let Y denote the number of games played. Assuming that the two teams are equally strong, determine the
Problem H (X ), H (Y ), H (Y X ) and H (X Y ).
values of 27 (The second law of thermodynamics) Let X = X1 , X2 , . . . be a stationary Markov chain.
Show that H (Xn X1 ) is monotone increasing (in spite of the fact that H (Xn ) does not change with n by
stationarity). Problem 3: Textbook, page 51, 2.36  Symmetric Relative Entropy Problem 28 (Properties of the binary entropy function) Deﬁne the function h(x) = −x log x −
Problem 4: Texbook, page 48, 2.21  Markov’s inequality for probabilities
(1 − x) log(1 − x), for x ∈ (0, 1), and h(0) = h(1) = 0. Show that h satisﬁes the following properties: Problem 5: Texbook, page 92, 4.10  Pairwise independence
• symmetric around 1/2; • continuous in every point of [0, 1]; 4 • strictly monotone increasing in [0, 1/2];
• strictly concave. 5 ...
View
Full
Document
This note was uploaded on 10/24/2011 for the course ELECTRICAL ECE 571 taught by Professor Kelly during the Spring '11 term at University of Illinois, Urbana Champaign.
 Spring '11
 Kelly

Click to edit the document details