1
Colorado State University, Ft. Collins
Fall 2008
ECE 516: Information Theory
Lecture 12 October 7, 2008
Recap:
6
Universal Source Coding (Chapter 13)
Arithmetic Coding
LempelZiv Coding
7
Channel Capacity (Chapter 7)
Channel
user
A
X
user
B
Y
p
(
X
)
p
(
Y

X
)
Source is a r.v.
X
with PMF
(
)
X
p
,
X
∈
x
The channel is a conditional PMF,
(
)
X
Y
p

,
X
∈
x
and
Y
∈
y
.
We will define “information” channel capacity
(
)
(
)
Y
X
I
C
x
p
;
max
=
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
2
Example of channel capacity:
1.
Noiseless binary channel
0
1
0
1
Y
=
X
(
)
(
)
(
)
(
)
(
)
(
)
X
H
X
X
I
Y
X
I
C
x
p
x
p
x
p
max
;
max
;
max
=
=
=
( )
⎟
⎠
⎞
⎜
⎝
⎛
=
2
1
,
2
1
x
p
and
1
=
C
bit
2.
Binary symmetric channel
0
1
0
1
1
p
1
p
p
p
(
)
(
)
(
)
(
)
( )
(
)
(
)
( )
(
)
(
)
(
)
(
)
p
H
p
H
Y
H
p
H
x
p
Y
H
x
X
Y
H
x
p
Y
H
X
Y
H
Y
H
Y
X
I
X
X
−
≤
−
=
−
=
=
−
=
−
=
∑
∑
=
=
1


;
1
0
1
0
But note when
( )
⎟
⎠
⎞
⎜
⎝
⎛
=
2
1
,
2
1
x
p
, we have
( )
⎟
⎠
⎞
⎜
⎝
⎛
=
2
1
,
2
1
y
p
and
(
)
1
=
Y
H
Therefore,
(
)
(
)
(
)
p
H
Y
X
I
C
x
p
−
=
=
1
;
max
Note,