Set Slide IV: Lossy Source Coding
Lossy Source Code Rate vs. Distortion for Memoryless Sources Rate vs. Distortion for Sources with Memory Universal Lossy Source Coding Shannon Lower Bound
EE477, Autumn 04-05, Lec. IV
1
Lossy Source Code
Assume X , Y are, respectively, the finite source and reconstruction alphabet. Assume a given single-letter distortion measure : X Y [0, ) which induces a block distortion: 1 (xn, y n) = (xi, yi). n i=1
n
EE477, Autumn 04-05, Lec. IV
2
Lossy Source Code (cont.)
Definition 1. A fixed-rate n-block code is a pair (Cn, n) where Cn Y n is the code-book and n : X n Cn. The rate of the block-code is given by 1 n n n n n log |Cn |. Y = Y (X ) = n (X ) will denote the reconstruction sequence when the n-block code is applied to the source sequence X n. The expected distortion is E(X n, Y n). Definition 2. A variable-rate code for n-blocks is a triple (Cn, n, ln) with Cn and n as in Definition 1. The code operates by mapping a source n-tuple X n into Cn via n, and then encoding the corresponding member of the code-book (denoted Y n in Definition 1) using a uniquely decodable binary code. Letting ln(X n) denote the associated length function, the rate of the 1 code is defined by E n ln(X n). The expected distortion is E(X n, Y n).
EE477, Autumn 04-05, Lec. IV
3
Rate Distortion Function
The Rate Distortion function associated with the random variable X is R(D) =
E(X,Y )D
min
I(X; Y )
EE477, Autumn 04-05, Lec. IV
4
Rate Distortion Theorem
Theorem 1. Let X = {Xi} be an i.i.d. source Xi X. [Direct:] For any > 0 and n sufficiently large there exists an n-block code with rate R(D) + and distortion D + . [Converse:] Any code-book Cn with rate R(D) has distortion D.
EE477, Autumn 04-05, Lec. IV
5
Strongly Typical Sequences and Rate Distortion
For xn X n let N (a|xn) = |{1 i n : xi = a}|. The strongly typical set TX is the set of all sequences xn X n satisfying 1.
1 N (a|xn) n
- PX (a)
|X |
for all a X with PX (a) > 0 and
2. N (a|xn) = 0 for all a X with PX (a) = 0. TX,Y is defined analogously. Lemma 1. Let {Yi} be i.i.d. Y . For xn TX 2-n(I(X;Y )+ where
1
1)
Pr (xn, Y n) TX,Y 2-n(I(X;Y )-
1)
,
0 as 0 and n .
EE477, Autumn 04-05, Lec. IV
6
Recalling Achievability Idea in Rate Distortion Theorem
Let (X, Y ) achieve the rate distortion function. Generate codewords in code-book independently with components i.i.d. Y By Lemma 1, for typical source realizations xn, probability that xn be jointly typical with, say, the first codeword is 2-nI(X;Y ). So, if we generate 2n(I(X;Y )+ ) codewords at least one of them will be jointly typical with the source with high probability Distortion between this codeword and source will be E(X, Y ) Recall, e.g., [Berger71], [Gallager68], [CT91, Sec. 13.6].
EE477, Autumn 04-05, Lec. IV
7
Remarkably Simple Converse
Fix a code-book with rate R and distortion D. R 1 H(Y n) n 1 I(X n; Y n) n 1 I(Xi; Yi) n i=1
n n
R(E(Xi, Yi))
i=1
1 R n
n
E(Xi, Yi)
i=1
R (D) .
EE477, Autumn 04-05, Lec. IV 8
Rate Distortion Function for Sources with Memory
The Rate Distortion function associated with X n is 1 I(X n; Y n). R(X , D) = minn E(X n ,Y )D n
n
The Rate Distortion function associated with the stationary source X is R(X, D) = lim R(X n, D) = inf R(X n, D).
n n
The existence of the limit, as well as the second equality, will be established in the HW exercise. You will also show that R(X, D) = inf{I(X; Y) : E(X0, Y0) D}, the infimum taken over jointly stationary processes consistent with the 1 distribution of X and I(X; Y) = lim n I(X n; Y n) denotes the mutual information rate.
EE477, Autumn 04-05, Lec. IV 9
Rate Distortion Theorem for Stationary Ergodic Sources
Theorem 2. Let X be stationary ergodic. [Direct:] For any > 0 and n sufficiently large there exists a code-book for n-blocks with rate R(X, D) + and distortion D + . [Converse:] Any code-book Cn with rate R(X, D) has distortion D.
EE477, Autumn 04-05, Lec. IV
10
Achievability Idea
Achievability of R(X1, D): Same as i.i.d. case. Achievability of R(X n, D): Similar to achievability of R(X1, D) by considering super alphabet X n Only problem is: "super process" may not be ergodic. But can be separated into, at most, n "ergodic modes". Letting Pn(y n|xn) achieve R(X n, D) and represent i the mode 1 n n n nR(X , D) = I(X ; Y ) Ii(X n; Y n) n i=1
n n i=1
R(X n, Di|i),
(notation Ii(X n; Y n) and R(X n, Di|i) to be explained in class) where n 1 Di = E[(X n, Y n)|i] so D = n i=1 Di.
EE477, Autumn 04-05, Lec. IV 11
Thus, for large L can find codes of block length nL attaining distortion essentially Di and rate R(X n, Di|i) on the i-th mode. Average distortion n n 1 1 is then D = n i=1 Di and average rate is n i=1 R(X n, Di|i) R(X n, D). Details can be found in [Berger71], [Gallager68].
EE477, Autumn 04-05, Lec. IV
12
Universal Lossy Source Coding
David L. Neuhoff, Paul C. Shields; Simplistic Universal Coding, IEEE Trans. Info. Theory, vol. IT-44, pp. 778 - 781, March 1998. [Ziv72] J. Ziv. Coding of sources with unknown statistics - part II: Distortion relative to a fidelity criterion. IEEE Trans. Inform. Theory, 18:389394, May 1972. [YK96] E. H. Yang and J. Kieffer. Simple universal lossy data compression schemes derived from the Lempel-Ziv algorithm. IEEE Trans. Inform. Theory, 42:239245, 1996. We follow [YK96]
EE477, Autumn 04-05, Lec. IV
13
Universal Lossy Source Coding (cont.)
A Simple Fixed-Rate Universal Lossy Coding Scheme: Let the code-book be given by ^ Bn = {y n X n : lLZ (y n) nR}. The encoding is done by giving the index of y n Bn that minimizes (xn, y n). Note universality of code-book w.r.t. source distribution and also w.r.t. distortion measure ! (1)
EE477, Autumn 04-05, Lec. IV
14
Universal Lossy Source Coding (cont.)
A Simple Fixed-Distortion Universal Lossy Coding Scheme: ^ Code-book: List the elements of X n in nondecreasing order of LZ codeword length. Encoding: Encode source n-block into the LZ codeword of the first word in code-book for which the resulting distortion is D.
EE477, Autumn 04-05, Lec. IV
15
Universality Result
Theorem 3. [YK96] For X stationary and ergodic E(X n, Bn) D(X, R) as n .
EE477, Autumn 04-05, Lec. IV
16
Proof of Theorem 3
Our proof will use the following lemma: ^ Lemma 2. Let m be a positive integer and : X m {1, 2, . . .} be a length function. Then there exists a sequence of positive real numbers { n} ^ tending to zero such that for all n and y n X n lLZ (y n) min
i+m-1 (yi ) ij(mod m),1in-m+1
1jm
+ nn.
Proof of Theorem 3: See white-board.
EE477, Autumn 04-05, Lec. IV
17
The Shannon Lower Bound
The Shannon lower bound we now derive will be key in our treatment of the compression-based approach to denoising. Let A be an alphabet where addition and subtraction of elements are well-defined. Let be a difference distortion measure (x, y) = (y - x). The m...