——————————————————————————————————————————————————
3.13, Cover and Thomas
Calculation of Typical Set
To clarify the notion of typical set
A
(
n
)
±
, and the smallest set of high probability
B
(
n
)
δ
,
we will calculate the set for a simple example. consider a sequence of i.i.d. binary random variables,
X
1
,X
2
,
···
n
,
where the probability that
X
i
= 1 is .6 (and therefore the probability that
X
i
= 0 is .4).
a. Calculate
H
(
X
).
Solution:
H
(
X
)=
±
x
∈{
0
,
1
}
p
(
x
)
·
log p
(
x
.
6
·
log .
6+
.
4
·
log .
4=
.
9710
b. With
n
=25and
±
=
.
1, which sequences fall in the typical set
A
(
n
)
±
?
Solution: Note that in general:
A
(
n
)
±
=
²
(
x
1
,x
2
,
n
)
∈
X
⇒
H
(
X
)
−
±
≤−
1
n
log p
(
x
1
2
,
n
)
≤
H
(
X
)+
±
³
For this problem:
A
(
n
)
±
=
²
x
n
∈{
0
,
1
}
n
⇒
H
(
X
)
−
±
1
n
log p
(
x
n
)
≤
H
(
X
±
³
=
²
x
25
0
,
1
}
25
⇒
.
9710
−
.
1
1
25
log p
(
x
25
)
≤
.
9710 +
.
1
³
=
²
x
25
0
,
1
}
25
⇒
.
8710
1
25
log
(
(
.
6)
k
·
(
.
4)
25
−
k
)
≤
1
.
0710
³
=
{
the sequences where k
,
the number of ones = 11
,
12
,
13
,
14
,
15
,
16
,
17
,
18
,
19
}
What is the probability of the typical set?
Solution: In Matlab: sum(binopdf(11:19,25,.6))=.9362
How many elements are their in the typical set?
Solution:
Elements
=
´
25
11
µ
+
´
25
12
µ
+
´
25
13
µ
+
´
25
14
µ
+
´
25
15
µ
+
´
25
16
µ
+
´
25
17
µ
+
´
25
18
µ
+
´
25
19
µ
=2
6
,
366
,
510
c. How many elements are there in the smallest set that has probability
.
9?
Solution: This will be done in two ways: First we compute this directly by adding the number of sequences
with
k
=12
,
13
,
25 including just enough of the sequences with 12 ones to equal
.
9 probability. We start
from the bottom since
.
6
>.
4
Hence, in MATLAB, we have
[
.
9
−
sum
(
binopdf
(13 : 25
,
25
,.
6))]
¶
·¸
¹
probability mass lef t in k
=12
/
(
.
6
(12)
∗
.
4
(13)
)
¶
·¸
¹
probability mass of one element
+
sum
(
NchooseK
(25
,
13 : 25))
¶
·¸
¹
total elements in k
=13
,
14
,
···
25
.
0458
e
+007
or we can use the formula

B
(
n
)
δ

=

B
(25)
.
1

·
nH
25
∗
.
9710
.
0283
e
+ 007
Question 1:
ECE 563, Fall 2011
Homework 3 Solution