huffman_min_variance

huffman_min_variance - CS 493: Algorithms for Massive Data...

Info iconThis preview shows pages 1–3. Sign up to view the full content.

View Full Document Right Arrow Icon

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: CS 493: Algorithms for Massive Data Sets Huffman and Arithmetic Coding DATE : Thursday, 2/7/2002 Scribe: Chi Zhang 1 Review of last class For message set S , s ∈ S has probability of p ( s ). Entropy of S is given as H ( S ) = X s ∈ S p ( s ) log 2 1 p ( s ) And log 2 1 p ( s ) is the self information of s. For any uniquely decodable code C for S , l a ( C ) ≥ H ( S ). Theorem 1.1 Given S , ∃ code C for S , such that l a ( C ) ≤ H ( S ) + 1 . Proof. Define l ( s ) = d log 2 1 p ( s ) e . X s ∈ S 2- l ( s ) = X s ∈ S 2-d log 2 1 p ( s ) e ≤ X s ∈ S p ( s ) = 1 From Kraft-McMillan inequality, ∃ prefix code C , l a ( C ) = X s ∈ S p ( s ) l ( s ) = X s ∈ S p ( s ) & log 2 1 p ( s ) ’ ≤ X s ∈ S p ( s ) log 2 1 p ( s ) + 1 ! = H ( S ) + 1 2 Huffman Code Given a set of messages with probabilities p 1 ≤ p 2 ≤ ... ≤ p n , the Huffman code tree is constructed by recursively combining subtrees: 1. Begin with n trees, each consists of a single node corresponding to one message word, with the weight of p i 2. Repeat until there is only one tree • pick two subtrees with smallest weights • combine them by adding a new node as root, and make the two trees its children. The weight of the new tree is the sum of the weight of two subtrees With a heap, each step of combining tree takes O (log n ) time, and the total time is O ( n log n ). 1 Lemma 2.1 Suppose C is the optimal code for S , p 1 , p 2 and l 1 , l 2 are the probabilities and code lengths of messages s 1 and s 2 , respectively. Then p 1 > p 2 ⇒ l 1 ≤ l 2 . Proof. Suppose p 1 > p 2 and l 1 > l 2 , we swap the code words for s 1 and s 2 , and get a new code C . The length of C is l a ( C ) = l a ( C )+ p 1 ( l 2- l 1 )+ p 2 ( l 1- l 2 ) = l a ( C )+( p 1- p 2 )( l 2- l 1 ) < l a ( C ). This contradicts the optimality of code C . Lemma 2.2 Without loss of generality, the two messages of smallest probability occur as siblings in the code tree for an optimal code. Proof. Given the code tree for an optimal code, we will show that it can always be modified without increasing the average code length so that the two smallest probability nodes are siblings. From Lemma 2.1, the smallest probability node must occur at the largest depth in the code tree. Note that the sibling of this node is also at the same depth. Now the sibling can be swapped with the second smallest probability node to obtain a code tree of the desired structure. This transformation does not increase the average code length....
View Full Document

This note was uploaded on 02/05/2012 for the course EE EE308 taught by Professor B.k.dey during the Spring '09 term at IIT Bombay.

Page1 / 5

huffman_min_variance - CS 493: Algorithms for Massive Data...

This preview shows document pages 1 - 3. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online