This preview shows pages 1–5. Sign up to view the full content.
1
Colorado State University, Ft. Collins
Fall 2008
ECE 516: Information Theory
Lecture 12 October 7, 2008
Recap:
6
Universal Source Coding (Chapter 13)
Arithmetic Coding
LempelZiv Coding
7
Channel Capacity (Chapter 7)
Channel
user
A
X
user
B
Y
p
(
X
)
p
(
Y

X
)
Source is a r.v.
X
with PMF
( )
X
p
,
X
∈
x
The channel is a conditional PMF,
( )
X
Y
p

,
X
∈
x
and
Y
∈
y
.
We will define “information” channel capacity
()
Y
X
I
C
x
p
;
max
=
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document 2
Example of channel capacity:
1.
Noiseless binary channel
0
1
0
1
Y
=
X
()
( )
( )
X
H
X
X
I
Y
X
I
C
x
p
x
p
x
p
max
;
max
;
max
=
=
=
⎟
⎠
⎞
⎜
⎝
⎛
=
2
1
,
2
1
x
p
and
1
=
C
bit
2.
Binary symmetric channel
0
1
0
1
1
p
1
p
p
p
()(
)
( )
() () ( )
() () ()
() ()
p
H
p
H
Y
H
p
H
x
p
Y
H
x
X
Y
H
x
p
Y
H
X
Y
H
Y
H
Y
X
I
X
X
−
≤
−
=
−
=
=
−
=
−
=
∑
∑
=
=
1


;
1
0
1
0
But note when
⎟
⎠
⎞
⎜
⎝
⎛
=
2
1
,
2
1
x
p
, we have
⎟
⎠
⎞
⎜
⎝
⎛
=
2
1
,
2
1
y
p
and
1
=
Y
H
Therefore,
( ) ( )
p
H
Y
X
I
C
x
p
−
=
=
1
;
max
Note,
3
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
p
C
3.
Binary erasure channel
0
1
0
1
1
α
1
α
α
α
e
()(
)
( )
() () ( )
() () ()
() ()
α
H
Y
H
H
x
p
Y
H
x
X
Y
H
x
p
Y
H
X
Y
H
Y
H
Y
X
I
X
X
−
=
−
=
=
−
=
−
=
∑
∑
=
=


;
1
0
1
0
()
( ) ( ) ( )
H
H
Y
H
Y
X
I
C
x
p
x
p
−
≤
−
=
=
3
log
max
;
max
But is this achievable by a
( )
x
p
? No.
Let
q
X
p
=
=
1
and
q
X
p
−
=
=
1
0
Then
() ( )
( ) ( )
q
q
H
Y
H
−
−
−
=
1
,
,
1
1
hard to interpret
Define
⎩
⎨
⎧
=
=
otherwise
,
0
,
1
e
Y
E
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document4
Then
()
(
)
( ) ( ) ( )
E
Y
H
E
H
Y
E
H
Y
H
E
Y
H


,
0
+
=
+
=
=
() () ( ) () ( ) ( )
∑
=
′
′
=
′
=
+
=
+
=
1
0


e
e
E
Y
H
e
E
p
H
E
Y
H
E
H
Y
H
α
( )
q
H
e
E
Y
H
e
=
′
=
⇒
=
′

0
0

1
=
′
=
⇒
=
′
e
E
Y
H
e
Then
( ) ( )
( ) (
)()
[ ] ( )
(
)
q
H
H
q
H
H
H
Y
This is the end of the preview. Sign up
to
access the rest of the document.
This note was uploaded on 03/17/2010 for the course ECE 516 taught by Professor Rocky during the Spring '08 term at Colorado State.
 Spring '08
 Rocky

Click to edit the document details