Lecture10

# Lecture10 - Colorado State University, Ft. Collins ECE 516:...

This preview shows pages 1–4. Sign up to view the full content.

1 Colorado State University, Ft. Collins Fall 2008 ECE 516: Information Theory Lecture 10 September 30, 2008 Recap: Theorem: (Extended Kraft Inequality) For any countable infinite set of codewords that form a prefix code, codeword lengths L , , 2 1 l l satisfy 1 i l i D Conversely, given any L , , 2 1 l l satisfying this inequality, we can construct a prefix code with these codeword lengths. Theorem: For any instantaneous D-ary code for a r.v. X, () X H L D with equality iff i l p D i = . Definition: A PMF is called D-adic with respect to D if each of the probabilities is equal to n D for some n . I.e., the PMF is such that i n i D p = . Shannon-Fano Codes Given the PMF of the source m p p , , 1 L , unconstraint optimum (non-integer) i D i D i p p l 1 log log ˆ = = Choose = i D i p l 1 log ˆ where ⎡ ⎤ x is the smallest integer larger than x . Theorem: Let * * 2 * 1 , , , m l l l L be the optimum codeword lengths for PMF ( ) x p in D-ary alphabet, then = i i i l p L * * satisfies

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
2 () ( ) 1 * + < X H L X H D D Theorem: The minium expected codeword length per symbol for a stationary random process satisfies X H L * Theorem: If the PMF is ( ) x p and we designed the Shannon-Fano code using ( ) x q , i.e., = x q x l 1 log , then () ( ) ( ) [ ] ( ) ( ) 1 || || + + < = + q p D p H X l E L q p D p H p Interpretation: The penalty is the distance between the PMFs. Kraft inequality for uniquely decodable codes Theorem: (McMillan) The codeword lengths of any uniquely decodable code must satisfy the Kraft inequality 1 i l i D Conversely, given a set of lengths that satisfy this inequality, then it is possible to construct a uniquely decodable code with these lengths. (Indeed, can construct pre- fix code)
3 Theorem: (McMillan) The codeword lengths of any uniquely decodable code must satisfy the Kraft inequality 1 i l i D Conversely, given a set of lengths that satisfy this inequality, then it is possible to construct a uniquely decodable code with these lengths. (Indeed, can construct pre- fix code) Proof: (By Karush) Recall, if a code is uniquely decodable then its extension is non-singular. Assume codeword lengths m l l l L 2 1 We want to show 1 i l i D We start by looking at k i l i D .

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
This is the end of the preview. Sign up to access the rest of the document.

## This note was uploaded on 03/17/2010 for the course ECE 516 taught by Professor Rocky during the Spring '08 term at Colorado State.

### Page1 / 12

Lecture10 - Colorado State University, Ft. Collins ECE 516:...

This preview shows document pages 1 - 4. Sign up to view the full document.

View Full Document
Ask a homework question - tutors are online