EE 250 Information Theory
Introduction to !
Information Measures
Ertem Tuncel
Information
What is information? !
How can we formalize and measure it?!
What gives me more information?
Learning the o
EE 250 Information Theory
Data Compression
Ertem Tuncel
Source Codes
Description: A code is a mapping from
the source alphabet to a sequence of 0s
and 1s:
Example: The canonical mapping.
and
Source
EE 250 Information Theory
Entropy Rates of!
Stochastic Processes
Ertem Tuncel
Stochastic Processes
Description: Recall that stochastic
processes are characterized by joint
PMFs of arbitrary size:
St
EE 250 Information Theory
Channel Coding and
Capacity
Ertem Tuncel
Communication Model
ENCODER
MODULATIO
N
PHYSICAL!
CHANNEL
DECODER
DEMODULATION
Communication Model
ENCODER
CHANNEL
DECODER
We usuall
EE 250 Information Theory
Typicality
Ertem Tuncel
Empirical Entropy
Consider the normalized log-probability
for an i.i.d. process:
by the weak law of large numbers.
Weakly Typical Sequences
A sequen
EE 250 Information Theory
Lossy Source Coding and
Rate-Distortion Theory
Ertem Tuncel
Lossy Source Coding
ENCODER
DECODER
Goal: Compress
with as little
distortion as possible.
Distortion: A measure