Lecture10 - Colorado State University, Ft. Collins ECE 516:...

Info iconThis preview shows pages 1–4. Sign up to view the full content.

View Full Document Right Arrow Icon
1 Colorado State University, Ft. Collins Fall 2008 ECE 516: Information Theory Lecture 10 September 30, 2008 Recap: Theorem: (Extended Kraft Inequality) For any countable infinite set of codewords that form a prefix code, codeword lengths L , , 2 1 l l satisfy 1 i l i D Conversely, given any L , , 2 1 l l satisfying this inequality, we can construct a prefix code with these codeword lengths. Theorem: For any instantaneous D-ary code for a r.v. X, () X H L D with equality iff i l p D i = . Definition: A PMF is called D-adic with respect to D if each of the probabilities is equal to n D for some n . I.e., the PMF is such that i n i D p = . Shannon-Fano Codes Given the PMF of the source m p p , , 1 L , unconstraint optimum (non-integer) i D i D i p p l 1 log log ˆ = = Choose = i D i p l 1 log ˆ where ⎡ ⎤ x is the smallest integer larger than x . Theorem: Let * * 2 * 1 , , , m l l l L be the optimum codeword lengths for PMF ( ) x p in D-ary alphabet, then = i i i l p L * * satisfies
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
2 () ( ) 1 * + < X H L X H D D Theorem: The minium expected codeword length per symbol for a stationary random process satisfies X H L * Theorem: If the PMF is ( ) x p and we designed the Shannon-Fano code using ( ) x q , i.e., = x q x l 1 log , then () ( ) ( ) [ ] ( ) ( ) 1 || || + + < = + q p D p H X l E L q p D p H p Interpretation: The penalty is the distance between the PMFs. Kraft inequality for uniquely decodable codes Theorem: (McMillan) The codeword lengths of any uniquely decodable code must satisfy the Kraft inequality 1 i l i D Conversely, given a set of lengths that satisfy this inequality, then it is possible to construct a uniquely decodable code with these lengths. (Indeed, can construct pre- fix code)
Background image of page 2
3 Theorem: (McMillan) The codeword lengths of any uniquely decodable code must satisfy the Kraft inequality 1 i l i D Conversely, given a set of lengths that satisfy this inequality, then it is possible to construct a uniquely decodable code with these lengths. (Indeed, can construct pre- fix code) Proof: (By Karush) Recall, if a code is uniquely decodable then its extension is non-singular. Assume codeword lengths m l l l L 2 1 We want to show 1 i l i D We start by looking at k i l i D .
Background image of page 3

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Image of page 4
This is the end of the preview. Sign up to access the rest of the document.

This note was uploaded on 03/17/2010 for the course ECE 516 taught by Professor Rocky during the Spring '08 term at Colorado State.

Page1 / 12

Lecture10 - Colorado State University, Ft. Collins ECE 516:...

This preview shows document pages 1 - 4. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online