1
Colorado State University, Ft. Collins
Fall 2008
ECE 516: Information Theory
Lecture 2 August 28, 2008
Recap
No class on Sep. 4 and Oct. 16.
0
Non-technical Remarks
Key objective: learning about how to formulate a research problem.
Follow Shannon’s soul, but not necessarily his method.
0.2
Some properties
Highly mathematical.
Theoretical understandings may seem to be useless.
Theoretical and Engineering researches.
2
Information Measures: Entropy, Relative Entropy, and Mutual
Information (Chapter 2)
2.1
Entropy
Define information of event
x
as
( )
( )
x
p
x
f
1
log
=
.
Definition: The entropy of a discrete r.v.
X
is
(
)
( )
( )
( )
( )
∑
∑
∈
∈
=
−
=
X
X
x
x
x
p
x
p
x
p
x
p
X
H
1
log
log
Lemma:
(
)
0
≥
X
H
Lemma:
(
)
(
)
X
aH
X
H
a
b
b
log
=

This
** preview**
has intentionally

**sections.**

*blurred***to view the full version.**

*Sign up*