Classical Information Theory
(Notes for Course on Quantum Computation and Information Theory. Sec. 11)
Robert B. Griffiths
Version of 17 Feb. 2003
References:
CT = Cover and Thomas,
Elements of Information Theory
(Wiley, 1991): chapters 2, 3,
and 8.
QCQI =
Quantum Computation and Quantum Information
by Nielsen and Chuang
(Cambridge, 2000), Secs. 11.1, 11.2
11
Classical Information Theory
11.1
Introduction
F
In order to discuss quantum computing and quantum cryptography in the presence of
noise, one needs to go beyond qualitative statements and introduce quantitative measures of
quantum information. At the present time this task is far from complete, and it isn’t clear
that we have the best approach.
•
The most useful quantitative quantum measures developed thus far are based on analo
gies with
classical
information theory, a welldeveloped subject with many applications to
real communication problems. So learning something about it is worthwhile even if eventu
ally we have to use something quite different to handle quantum information.
◦
The material below is a rapid survey of certain ideas in classical information which
look like they are of use in the quantum domain.
•
CT is a standard reference for (classical) information theory. QCQI provides a very
abbreviated introduction to the subject with a stress on mathematical properties. The notes
below are intended to be a more intuitive introduction to the subject, with proofs and lots
of details left to QCQI and CT.
11.2
Shannon Entropy
F
Suppose we have a certain message we want to transmit from location
A
to location
B
. What resources are needed to get it from here to there? How long will it take if we have
a channel with a capacity of
c
bits per second? If transmission introduces errors, what do we
1
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
do about them? These are the sorts of questions which are addressed by information theory
as developed by Shannon and his successors.
•
There are, clearly,
N
= 2
n
messages which can be represented by a string of
n
bits;
e.g., 8 messages 000, 001,. . . 111 for
n
= 3.
Hence if we are trying to transmit one of
N
distinct messages it seems sensible to define the amount of information carried by a single
message to be log
2
N
bits, which we hereafter abbreviate to log
N
(in contrast to ln
N
for
the natural logarithm).
◦
Substituting some other base for the logarithm merely means multiplying log
2
N
by
a constant factor, which is, in effect, changing the units in which we measure information.
E.g., ln
N
nits in place of log
N
bits.
F
A key observation is that if we are in the business of
repeatedly
transmitting messages
from a collection of
N
messages, and if the messages can be assigned a nonuniform
probability
distribution
, then it is, on average, possible to use fewer than log
N
bits per message in order
to transmit them, or to store them.
This is the end of the preview.
Sign up
to
access the rest of the document.
 Spring '09
 B.K.Dey
 Information Theory, Probability distribution, Bob Y

Click to edit the document details