Compression Data Notes COMP 4923/5113 Chapters 4 5
Jim Diamond Jodrey School of Computer Science Acadia University
Chapter 4
Chapter 4
40
Statistical Compression: Once More, With Feeling
Human compresses a sequence of input symbols into a sequence of bits use same probabilities and codes as example from Chapter 3 e.g., BAD 100111
Imagine a binary point in front: .1001112
BAD is assigned a number as follows: code(B ) 22 + code(A) 23 + code(D ) 26
= 102 22 + 0 23 + 1112 26 = 0.60937510
Every string compresses into some number x [0, 1) Problem: Human coding achieves the entropy only if all symbols have probabilities that are inverse powers of 2 in some other cases (pmax large), Human codes are quite poor compared to the entropy of the data source
By doing appropriate arithmetic calculations, we can overcome this problem
Jim Diamond, Jodrey School of Computer Science, Acadia University
Chapter 4
41
Arithmetic Compression
Any message encodes into a number x [0, 1) all strings starting with A are in [0, 0.1) all strings starting with B are in [0.1, 0.11) all strings starting with C are in [0.11, 0.111) all strings starting with D are in [0.111, 1)
[0
Thus the unit interval is partitioned according to the rst symbol in the message:
0.1) [0.1
A B
0.11) [.11
C
.111) [.111
D
1)
Jim Diamond, Jodrey School of Computer Science, Acadia University
Chapter 4
42
Arithmetic Compression: 2
The resulting interval is sub-partitioned according to the second symbol, and so on:
A A D B C D
Arithmetic code for a message is dened by interval start and width note: the arithmetic code is for the whole message, not for individual symbols!
Unlikely strings are assigned narrow intervals more bits required to represent them
Advantage over Human: interval widths need not be powers of 1/2! this causes a disadvantage: actually computing the intervals can be messy
Jim Diamond, Jodrey School of Computer Science, Acadia University
Chapter 4
43
Arithmetic Compression: Implementation Issues
See textbook for algorithms Do we need innite-precision arithmetic to compute intervals? quantities added to (a) start of interval and (b) width of interval get smaller and smaller after a while some number of the leading digits will never change, so they can be output tricky if interval overlaps 0.5
What if probabilities are repeating fractions in binary? (a) round somehow while maintaining
P (ai) = 1 ??
(b) estimate them in a way to avoid this in rst place how? (good exam question)
Jim Diamond, Jodrey School of Computer Science, Acadia University
Chapter 4
44
Arithmetic Compression: Implementation Issues 2
What is the actual compressor output? one approach: use lower endpoint another approach: use midpoint in fact, you can just pick any (shortest) number in the nal interval to represent input
Decoder must know how many bits it received (problem in last byte!) one solution: add an EOF symbol to the set of possible input symbol assign it an extremely small probability this gives it a relatively long associated bit string, but that only occurs once
another solution: send count of input symbols before compressed data online problem?
Jim Diamond, Jodrey School of Computer Science, Acadia University
Chapter 4
45
Arithmetic Compression: Implementation Issues 3
Both encoder and decoder need to know model of data i.e., what is P (ai ) (ai A)? and this may change after every symbol is encoded!
If encoder has to send the model to the decoder, there may be considerable overhead If encoder and decoder share a particular data model, they may not be applicable to a variety of types of data
Jim Diamond, Jodrey School of Computer Science, Acadia University
Chapter 4
46
Arithmetic Compression vs. Human
Generating and outputting arithmetic codes individually for each symbol (rather than one code for a sequence) gives worse results than Human Generating codes for (long) sequences: Human: too many combinations for length-m sequences: |A|m arithmetic compression: no problem (except for implementation details!)
If many symbols, distribution not too skewed, Human does OK extra complexity of arithmetic coding may not be worth it
Adaptive compression: Human: need to update or rebuild tree arithmetic: just change current interval sizes
Jim Diamond, Jodrey School of Computer Science, Acadia University
Chapter 4
47
Adaptive Compression
For Human coding, model of data is a set of probabilities which determines a set of bit strings For arithmetic compression, model of data is a set of probabilities which determines a set of intervals How is model dened? statistics from entire input le statistics from all symbols seen so far statistics from rst n input symbols statistics from most recently seen m symbols recompute statistics every k symbols combine this with either of previous two
Jim Diamond, Jodrey School of Computer Science, Acadia University
Chapter 4
48
Adaptive Compression: 2
Recomputing model frequently not necessarily desirable compute intensive more dicult to implement correctly
What should initial model be? it could contain all possible symbols, with some initial PDF guess it might take a long time for probabilities to converge to representative values
it could contain no symbols at all add them as needed need escape codes to signify that next bits represent a new symbol, not regular compressed data
Jim Diamond, Jodrey School of Computer Science, Acadia University
Chapter 4
49
Adaptive Human (Sketch)
Each leaf in the Human tree keeps a count of how many times its symbol has been seen Each internal node in the tree keeps a count of the sum of all leaves below it To encode the next symbol s: if the symbol has been seen, encode it with its current code otherwise, use an escape code
After encoding the symbol: if there is a node for s in the Human tree, increment it; otherwise nd the node N in the tree with smallest count, create a new node N for s with count 1, create a new internal node N and make N and N sons of N
Verify the tree is still valid: because counts have changes, it may need to be rebalanced See text (Chapter 3) for details
Jim Diamond, Jodrey School of Computer Science, Acadia University
Chapter 4
50
Adaptive Human: 2
If statistics are generated using previous m symbols, counts in nodes will sometimes decrease tree re-balancing also needed here may even need to delete tree nodes
Interesting idea. Useful in practice?
Jim Diamond, Jodrey School of Computer Science, Acadia University
Chapter 4
51
Adaptive Human: Examples
Example: Human coding of a 41082428 byte sound le (16-bit stereo) non-adaptive Human: 32191987 (78.6% compression) adaptive Human: 32147909 (78.3% compression) dierence very, very small in this example
Example: Human coding of a 31978 byte sound le (8-bit mono) non-adaptive Human: 25280 (79.0% compression) adaptive Human: 24933 (78.0% compression)
Better: predictive (and adaptive, for large sound le) linear prediction: 26534322 (64.6% compression) quadratic prediction: 27317512 (66.5% compression)
Jim Diamond, Jodrey School of Computer Science, Acadia University
Chapter 4
52
Adapting To Windows of Data
Re-start compression every k symbols (or maybe when l bits have been output) Disadvantages may have overhead of transmitting model (statistics) adaptive: lose (statistical) from info recent data
Advantages if data changing quickly, may get better overall compression if bit(s) corrupted in transmission, only the data to the next window is corrupted (if decoder can nd window boundary!) e.g., video data: a full frame is occasionally sent, and next few frames are encoded by sending dierences between adjacent frames (very roughly speaking) if there is an error in a full frame, the next few frames probably also will have errors
Jim Diamond, Jodrey School of Computer Science, Acadia University
Chapter 5
Jim Diamond, Jodrey School of Computer Science, Acadia University
Chapter 5
53
Dictionary Techniques
Human (and Shannon-Fano) compressors input xed-size symbols and generate variable-size outputs Alternative: generate xed-size outputs, each from a variable number of input symbols Example: English text some words occur frequently (the) some strings occur frequently as sub-words (th, gh, ...)
Idea of dictionary techniques: have some dictionary (list of strings) that the compressor and decompressor know if the compressor sees a string in the dictionary, it outputs a code referring to this string, rather than the string itself
Jim Diamond, Jodrey School of Computer Science, Acadia University
Chapter 5
54
Dictionary Techniques 2
Example: suppose dictionary is 0: the 1: computer 2: string 3: poodle and the compressor input is the poodle ate two strings
then the compressor output could be 0 3 ate two 2 s dictionary index can be represented with 2 bits (in this case) literals can be represented in (e.g.) ISO-8859-1
Need some way to distinguish a dictionary index from literal data overhead!
How big is the dictionary? Should it stay constant? Where does it come from?
Jim Diamond, Jodrey School of Computer Science, Acadia University
Chapter 5
55
Dictionary Techniques: Ziv and Lempel (1977) (LZ77, aka LZ1)
Idea: sliding window moves through text to be compressed two parts: text that has been compressed, text to be compressed ts: text that has been compressed, text to be compr
Compressor: look in search buer (on left) for beginning of look-ahead buer (on right) the t matches the last t of that te doesnt match t ; look back further for longer matches text t at beginning of look-ahead buer is longest match compressor outputs (31,6,o) i.e., index from end of search buer, match length, next char why do we need next char?
Window then slides ahead 7 chars More exible dictionary structure than previous slide
Jim Diamond, Jodrey School of Computer Science, Acadia University
Chapter 5
56
LZ78 (aka LZ2): New and Improved
Various people proposed improvements to LZ77, but there are some problems size of look-ahead buer trade-o: frequent words may not be close enough together search time, bits required to represent oset & length increase with buer size
Jim Diamond, Jodrey School of Computer Science, Acadia University
Chapter 5
56
LZ78 (aka LZ2): New and Improved
Various people proposed improvements to LZ77, but there are some problems size of look-ahead buer trade-o: frequent words may not be close enough together search time, bits required to represent oset & length increase with buer size
LZ78: dictionary starts with only one entry (the null string), then grows as strings are added to it At each step match (in dictionary) as much of uncompressed text as possible output corresponding dictionary index plus next char add that text + unmatched symbol to dictionary
<<cut to demo>>
Jim Diamond, Jodrey School of Computer Science, Acadia University
Chapter 5
57
LZ78: Issues
Should dictionary increase without bound? If not should it stay static at some upper limit, or should entries be removed according to some rule?
How should dictionary be represented for ecient compression? Should length of dictionary index grow as dictionary grows?
Jim Diamond, Jodrey School of Computer Science, Acadia University
Chapter 5
58
Variations on LZ77: LZSS et al
Output tokens with 2 elds, not 3 omit the next char eld (how?? YA good exam question)
Use circular buer to store sliding window save CPU time moving chars around
Hold search buer in BST see text for other possibilities for speeding searching
Other variations: gzip, pkzip, ... these follow LZ with Human (see text)
Jim Diamond, Jodrey School of Computer Science, Acadia University
Chapter 5
59
Variations on LZ78: LZW
I...