This preview shows pages 1–3. Sign up to view the full content.
This preview has intentionally blurred sections. Sign up to view the full version.View Full Document
Unformatted text preview: CS 493: Algorithms for Massive Data Sets Huffman and Arithmetic Coding DATE : Thursday, 2/7/2002 Scribe: Chi Zhang 1 Review of last class For message set S , s ∈ S has probability of p ( s ). Entropy of S is given as H ( S ) = X s ∈ S p ( s ) log 2 1 p ( s ) And log 2 1 p ( s ) is the self information of s. For any uniquely decodable code C for S , l a ( C ) ≥ H ( S ). Theorem 1.1 Given S , ∃ code C for S , such that l a ( C ) ≤ H ( S ) + 1 . Proof. Define l ( s ) = d log 2 1 p ( s ) e . X s ∈ S 2- l ( s ) = X s ∈ S 2-d log 2 1 p ( s ) e ≤ X s ∈ S p ( s ) = 1 From Kraft-McMillan inequality, ∃ prefix code C , l a ( C ) = X s ∈ S p ( s ) l ( s ) = X s ∈ S p ( s ) & log 2 1 p ( s ) ’ ≤ X s ∈ S p ( s ) log 2 1 p ( s ) + 1 ! = H ( S ) + 1 2 Huffman Code Given a set of messages with probabilities p 1 ≤ p 2 ≤ ... ≤ p n , the Huffman code tree is constructed by recursively combining subtrees: 1. Begin with n trees, each consists of a single node corresponding to one message word, with the weight of p i 2. Repeat until there is only one tree • pick two subtrees with smallest weights • combine them by adding a new node as root, and make the two trees its children. The weight of the new tree is the sum of the weight of two subtrees With a heap, each step of combining tree takes O (log n ) time, and the total time is O ( n log n ). 1 Lemma 2.1 Suppose C is the optimal code for S , p 1 , p 2 and l 1 , l 2 are the probabilities and code lengths of messages s 1 and s 2 , respectively. Then p 1 > p 2 ⇒ l 1 ≤ l 2 . Proof. Suppose p 1 > p 2 and l 1 > l 2 , we swap the code words for s 1 and s 2 , and get a new code C . The length of C is l a ( C ) = l a ( C )+ p 1 ( l 2- l 1 )+ p 2 ( l 1- l 2 ) = l a ( C )+( p 1- p 2 )( l 2- l 1 ) < l a ( C ). This contradicts the optimality of code C . Lemma 2.2 Without loss of generality, the two messages of smallest probability occur as siblings in the code tree for an optimal code. Proof. Given the code tree for an optimal code, we will show that it can always be modified without increasing the average code length so that the two smallest probability nodes are siblings. From Lemma 2.1, the smallest probability node must occur at the largest depth in the code tree. Note that the sibling of this node is also at the same depth. Now the sibling can be swapped with the second smallest probability node to obtain a code tree of the desired structure. This transformation does not increase the average code length....
View Full Document
This note was uploaded on 02/05/2012 for the course EE EE308 taught by Professor B.k.dey during the Spring '09 term at IIT Bombay.
- Spring '09