Chapter 1
The Science of Information
Raymond W. Yeung 2010 Department of Information Engineering The Chinese University of Hong Kong
Information Theory
Founded by Claude E. Shannon (1916-2002) The Mathematical Theory of Communication, 1948 Study fundame

Chapter 5
Weak Typicality
Raymond W. Yeung 2012
Department of Information Engineering
The Chinese University of Hong Kong
The Notion of Typical Sequences
Toss a fair coin n times.
If the outcome is head approximately half of the time, the sequence of
o

Chapter 4
Zero-Error Data Compression
Raymond W. Yeung 2012
Department of Information Engineering
The Chinese University of Hong Kong
4.1 The Entropy Bound
Denition 4.1 A D-ary source code C for a source random variable X is a
mapping from X to D , the s

Chapter 3
The I-Measure
Raymond W. Yeung 2010 Department of Information Engineering The Chinese University of Hong Kong
An Example
H ( X1 ,X 2) H ( X1 X 2) H ( X 2 X1)
H ( X1) I ( X1; X 2)
H (X 2)
Substitution of Symbols
H/I , ; | is some signed measure

Chapter 9
The Blahut-Arimoto
Algorithms
Raymond W. Yeung 2012
Department of Information Engineering
The Chinese University of Hong Kong
Single-Letter Characterization
For a DMC p(y |x), the capacity
C = max I (X ; Y ),
r (x)
where r(x) is the input dist

Chapter 6
Strong Typicality
Raymond W. Yeung 2012
Department of Information Engineering
The Chinese University of Hong Kong
6.1 Strong AEP
Setup
cfw_X k , k
1, Xk i.i.d. p(x).
X denotes generic r.v. with entropy H (X ) < .
|X | <
6.1 Strong AEP
Denit