Chapter 2
Sphere Packing and
Shannon’s Theorem
In the first section we discuss the basics of block coding on the
m
ary symmetric
channel. In the second section we see how the geometry of the codespace can
be used to make coding judgements. This leads to the third section where we
present some information theory and Shannon’s basic Channel Coding Theorem.
2.1
Basics of block coding on the
m
SC
Let
A
be any finite set. A
block code
or
code
, for short, will be any nonempty
block code
subset of the set
A
n
of
n
tuples of elements from
A
. The number
n
=
n
(
C
) is
the
length
of the code, and the set
A
n
is the
codespace
. The number of members
length
codespace
in
C
is the
size
and is denoted

C

. If
C
has length
n
and size

C

, we say that
size
C
is an (
n,

C

)
code
.
(
n,

C

)
code
The members of the codespace will be referred to as
words
, those belonging
words
to
C
being
codewords
. The set
A
is then the
alphabet
.
codewords
alphabet
If the alphabet
A
has
m
elements, then
C
is said to be an
m
ary code
. In
m
ary code
the special case

A

=2 we say
C
is a
binary
code and usually take
A
=
{
0
,
1
}
binary
or
A
=
{
1
,
+1
}
.
When

A

=3 we say
C
is a
ternary
code and usually take
ternary
A
=
{
0
,
1
,
2
}
or
A
=
{
1
,
0
,
+1
}
. Examples of both binary and ternary codes
appeared in Section 1.3.
For a discrete memoryless channel, the Reasonable Assumption says that a
pattern of errors that involves a small number of symbol errors should be more
likely than any particular pattern that involves a large number of symbol errors.
As mentioned, the assumption is really a statement about design.
On an
m
SC(
p
) the probability
p
(
y

x
) that
x
is transmitted and
y
is received
is equal to
p
d
q
n

d
, where
d
is the number of places in which
x
and
y
differ.
Therefore
Prob(
y

x
) =
q
n
(
p/q
)
d
,
15
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
16
CHAPTER 2.
SPHERE PACKING AND SHANNON’S THEOREM
a decreasing function of
d
provided
q > p
. Therefore the Reasonable Assumption
is realized by the
m
SC(
p
) subject to
q
= 1

(
m

1)
p > p
or, equivalently,
1
/m > p .
We interpret this restriction as the sensible design criterion that after a symbol
is transmitted it should be more likely for it to be received as the correct symbol
than to be received as any particular incorrect symbol.
Examples.
(i) Assume we are transmitting using the the binary Hamming code
of Section 1.3.3 on BSC(
.
01). Comparing the received word 0011111 with
the two codewords 0001111 and 1011010 we see that
p
(0011111

0001111) =
q
6
p
1
≈
.
009414801
,
while
p
(0011111

1011010) =
q
4
p
3
≈
.
000000961 ;
therefore we prefer to decode 0011111 to 0001111.
Even this event is
highly unlikely, compared to
p
(0001111

0001111) =
q
7
≈
.
932065348
.
(ii) If
m
= 5 with
A
=
{
0
,
1
,
2
,
3
,
4
}
6
and
p
=
.
05
<
1
/
5 =
.
2, then
q
= 1

4(
.
05) =
.
8; and we have
p
(011234

011234) =
q
6
=
.
262144
and
p
(011222

011234) =
q
4
p
2
=
.
001024
.
For
x
,
y
∈
A
n
, we define
d
H
(
x
,
y
) = the number of places in which
x
and
y
differ.
This number is the
Hamming distance
between
x
and
y
. The Hamming distance
Hamming distance
is a genuine metric on the codespace
A
n
. It is clear that it is symmetric and
that d
H
(
x
,
y
) = 0 if and only if
x
=
y
. The Hamming distance d
H
(
x
,
y
) should
be thought of as the number of errors required to change
x
into
y
(or, equally
well, to change
y
into
x
).
This is the end of the preview.
Sign up
to
access the rest of the document.
 Spring '11
 NA
 Coding theory, Hamming Code, Minimum distance, SHANNON'S

Click to edit the document details