EE 376A/Stat 376A
Information Theory
Prof. T. Weissman
Thursday, March 23, 2006
Final
1. (
20 points
)
Three Shannon codes
Let
{
U
i
}
i
≥
1
be a stationary 1storder Markov source whose alphabet size is
r
. Note that
the stationarity property implies that
P
(
u
i
)
,P
(
u
i

u
i

1
) does not depend on
i
. Assume
that

log
P
(
u
i
) and

log
P
(
u
i

u
i

1
) are integers for all (
u
i
,u
i

1
). Three students
decided to use the Shannon coding scheme, based on diﬀerent pmfs as follows:
•
Alice suggested to code each symbol separately, i.e., code by using the pmf
{
P
(
u
i
)
}
.
•
Bob suggested to code in pairs by using the joint pmf
{
P
(
u
i
,u
i
+1
)
}
,i
= 1
,
3
,
5
,...
.
In other words, ﬁrst code (
U
1
,U
2
), then code (
U
3
,U
4
), and so on.
•
Charlie suggested to code each symbol given the previous symbol by using the
conditional pmf
{
P
(
u
i

u
i

1
)
}
. In other words, ﬁrst code
U
1
, then code
U
2
given
U
1
, then code
U
3
given
U
2
, and so on.
(a) If the source is memoryless, compare the expected codeword length per symbol,
i.e.,
1
n
E‘
(
U
n
), of each scheme, assuming
n >
2 is even.
(b) Repeat Part (a) for the case where the source has memory.
Solutions
(a) Because the source is memoryless Alice’s and Charlie’s coding schemes are identical.
In addition it is given that

log
P
(
u
i
) is an integer and hence Shanon coding for
block length 1 has an average length of
H
(
U
1
) =
∑
u
1
P
(
u
1
)log(
P
(
u
1
)). Bob’s coding
scheme is over pairs that their log of the PMF is also an integer, and therefore the
average length of his code is
1
2
H
(
U
1
,U
2
) and because of the memoryless property
1
2
H
(
U
1
,U
2
) =
H
(
U
1
). The conclusion is that all three codes have the same average
length.
(b) Alice’s Average code is
1
n
E‘
(
U
n
) =
1
n
"
n
X
i
=1
X
u
i
P
(
u
i
)log
P
(
u
i
)
#
=
H
(
U
1
)
1