EE 376A
Information Theory
Prof. T. Weissman
Friday, Feb 19, 2010
Homework Set #6
(Due: 5pm Friday, Feb. 26, 2010)
1.
Channel capacity with cost constraint
There are applications in which some chan
nel input symbols are more costly than others.
Letting Λ :
X  7→ R
denote the
“cost function”.
The per symbol cost of the transmitted sequence
X
n
is Λ(
X
n
) =
1
n
∑
n
i
=1
Λ(
X
i
). The expected per symbol cost of the code is then given by
EΛ(
X
n
) = 2

nR
2
n
R
X
k
=1
Λ(
x
n
(
k
))
,
where
x
n
(
k
) is the
k
th codeword in the code book.
Definition:
A rate
R
is achievable at cost Γ, if
∀
>
0 there exists
n
and a channel
code of blocklength n, rate
≥
R

,
E
Λ(
X
n
)
≤
Γ+
and probability of error
P
(
n
)
e
≤
.
Definition:
The capacitycost function of the channel is defined as
C
(Γ) = sup
{
R
:
R
is achievable at costΓ
}
.
In words,
C
(Γ) is the maximum rate of reliable communication when restricting the
transmission to an expected cost that does not exceed Γ per channel use.
(a) Prove that for a given
P
Y

X
(
y

x
),
I
(
X
;
Y
) is a concave function of
P
X
(
x
).
Hint:
First show that if
f
:
R
n
→ R
, is a convex function, then for any
n
×
m
realvalued matrix
A
,
g
(
X
)
,
f
(
AX
),
g
:
R
m
→ R
, is also a convex function,
i.e. linear transformation preserves convexity.
(b) Consider the following expression
C
(
I
)
(Γ) =
max
EΛ(
X
)
≤
Γ
I
(
X
;
Y
)
where the maximum on the right hand side is over all random variables X at the
channel input that satisfy the indicated cost constraint. Prove that
C
(
I
)
(Γ) is a
nondecreasing concave function in Γ. You may use the result from Part (a).
(c) The converse:
C
(Γ)
≤
C
(
I
)
(Γ).
For any sequence of schemes of rate
R
, EΛ(
X
n
)
≤
Γ, and vanishing probability of
error:
1
nR
=
H
(
M
)
=
I
(
M
;
Y
n
) +
H
(
M

Y
n
)
(
a
)
≤
I
(
M
;
Y
n
) +
n
n
(
b
)
≤
I
(
X
n
;
Y
n
) +
n
n
(
c
)
≤
H
(
Y
n
)

n
X
i
=1
H
(
Y
i

X
i
) +
n
n
(
d
)
≤
n
X
i
=1
I
(
X
i
;
Y
i
) +
n
n
(
e
)
≤
n
X
i
=1
C
(
I
)
(EΛ(
X
i
)) +
n
n
(
f
)
≤
nC
(
I
)
(EΛ(
X
n
)) +
n
n
(
g
)
≤
nC
(
I
)
(Γ) +
n
n
,
where
n
→
0 as
n
→ ∞
.
Provide explanations for inequalities (a) – (g).
Solutions:
(a) First we show that if
f
:
R
n
→ R
, is a convex function, then for any
n
×
m
realvalued matrix
A
,
g
(
X
)
,
f
(
AX
),
g
:
R
m
→ R
, is also a convex function.
Let 0
≥
λ
≥
1,
g
(
λX
1
+ (1

λ
)
X
2
) =
f
(
λAX
1
+ (1

λ
)
AX
2
)
,
≤
λf
(
AX
1
) + (1

λ
)
f
(
AX
2
)
,
=
λg
(
X
1
) + (1

λ
)
g
(
X
2
)
.
Consequently, linear transformation preserves convexity, equivalently concavity,
of functions. Now note that
I
(
X
;
Y
) =
H
(
Y
)

H
(
Y

X
)
=
H
(
Y
)

X
x
∈X
H
(
Y

X
=
x
)
P
(
X
=
x
)
.
(1)
But since
P
(
Y

X
) is fixed,
H
(
Y

X
=
x
) does not depend on
P
(
X
). Therefore,
H
(
Y

X
You've reached the end of your free preview.
Want to read all 8 pages?
 Spring '10
 sd
 Derivative, γ, Convex function, Concave function