This preview shows pages 1–3. Sign up to view the full content.
EE 376B/Stat 376B
Handout #11
Information Theory
Thursday, April 27, 2006
Prof. T. Cover
Solutions to Homework Set #3
1.
Maximum entropy discrete processes.
(a) Find the maximum entropy rate binary stochastic process
{
X
i
}
∞
i
=
∞
, X
i
∈ {
0
,
1
}
,
satisfying Pr
{
X
i
=
X
i
+1
}
=
1
3
, for all
i
.
(b) What is the resulting entropy rate?
Solution: Maximum entropy discrete processes
Our ﬁrst hope may be an i.i.d. Bern(
p
) process that is consistent with the constraint.
Unfortunately, there is no such process. (Check!) However, we can still construct
an independent
nonidentically
distributed sequence of Bernoulli r.v.’s, such that the
entropy rate exists, and the constraints are met. (For example,
X
i
∼
Bern(1) for odd
i
and
X
i
∼
Bern(1
/
3) for even
i
.) This process does not yield the maximum entropy
rate.
This problem is, in fact, a discrete (or more precisely, binary) analogue of Burg’s
maximum entropy theorem and we can obtain the maximum entropy process from a
similar argument.
(a) Let
X
i
be any binary process satisfying the constraint Pr
{
X
i
=
X
i
+1
}
= 1
/
3. Let
Z
i
be a ﬁrst order stationary Markov chain, that stays at 0 with probability 1/3,
jumps to 1 with probability 2/3, and vice versa. This process obviously meets the
constraint. With a slight abuse of notation, we have
H
(
X
i

X
i

1
) =
E
H
(
X
i

X
i

1
=
x
)
=
E
H
(Pr(
X
i
=
x

X
i

1
=
x
))
≤
H
(
E
Pr(
X
i
=
x

X
i

1
=
x
))
=
H
(1
/
3)
=
H
(
Z
i

Z
i

1
)
,
where the inequallity follows from the concavity of the binary entropy function.
1
This preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentSince
H
(
X
1
)
≤
1 =
H
(
Z
1
),
H
(
X
1
,...,X
n
) =
H
(
X
1
) +
n
X
i
=2
H
(
X
i

X
i

1
)
≤
H
(
X
1
) +
n
X
i
=2
H
(
X
i

X
i

1
)
≤
H
(
Z
1
) +
n
X
i
=2
H
(
Z
i

Z
i

1
)
=
H
(
Z
1
,...,Z
n
)
,
whence
{
Z
i
}
is the maximum entropy process under the given constraint.
(b) The maximum entropy rate
This is the end of the preview. Sign up
to
access the rest of the document.
 Spring '05
 TomCover

Click to edit the document details