lecture Recap: 2 CS276A
Information Retrieval
Stemming, tokenization etc. Faster postings merges Phrase queries
Lecture 3
This lecture
Index compression Space estimation
Corpus size for estimates
Consider n = 1M documents, each with about L=1K terms. Avg 6 bytes/term incl spaces/punctuation
6GB of data.
Say there are m = 500K distinct terms among these.
Dont build the matrix
500K x 1M matrix has half-a-trillion 0s and 1s. But it has no more than one billion 1s.
matrix is extremely sparse.
Where do we pay in storage?
Doc # Freq 2 2 1 2 1 1 2 1 1 2 1 1 2 1 1 2 1 2 2 1 2 2 2 1 2 2 1 1 1 1 1 1 2 1 1 1 2 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1
So we devised the inverted index
Devised query processing for it
Where do we pay in storage?
Terms
Term N docs Tot Freq ambitious 1 1 be 1 1 brutus 2 2 capitol 1 1 caesar 2 3 did 1 1 enact 1 1 hath 1 1 I 1 2 i' 1 1 it 1 1 julius 1 1 killed 1 2 let 1 1 me 1 1 noble 1 1 so 1 1 the 2 2 told 1 1 you 1 1 was 2 2 with 1 1
Pointers
1
Storage analysis
First will consider space for postings pointers Basic Boolean index only
Devise compression schemes
Pointers: two conflicting forces
A term like Calpurnia occurs in maybe one doc out of a million - would like to store this pointer using log2 1M ~ 20 bits. A term like the occurs in virtually every doc, so 20 bits/pointer is too expensive.
Prefer 0/1 vector in this case.
Then will do the same for dictionary No analysis for positional indexes, etc.
Postings file entry
Store list of docs containing a term in increasing order of doc id.
Brutus: 33,47,154,159,202
Variable encoding
For Calpurnia, will use ~20 bits/gap entry. For the, will use ~1 bit/gap entry. If the average gap for a term is G, want to use ~log2G bits/gap entry. Key challenge: encode every integer (gap) with ~ as few bits as needed for that integer.
Consequence: suffices to store gaps.
33,14,107,5,43
Hope: most gaps encoded with far fewer than 20 bits.
codes for gap encoding (Elias)
Length Offset
codes for gap encoding
e.g., 9 represented as <1110,001>. 2 is represented as <10,1>. Exercise: does zero have a code? Encoding G takes 2 log2G +1 bits.
codes are always of odd length.
Represent a gap G as the pair <length,offset> length is in unary and uses log2G +1 bits to specify the length of the binary encoding of offset = G - 2log2G in binary. Recall that the unary encoding of x is a sequence of x 1s followed by a 0.
2
Exercise
Given the following sequence of coded gaps, reconstruct the postings sequence:
What weve just done
Encoded each gap as tightly as possible, to within a factor of 2. For better tuning (and a simple analysis) - need a handle on the distribution of gap values.
1110001110101011111101101111011
From these decode and reconstruct gaps, then full postings.
Zipfs law
The kth most frequent term has frequency proportional to 1/k. Use this for a crude analysis of the space used by our postings file pointers.
Not yet ready for analysis of dictionary space.
Zipfs law log-log plot
Rough analysis based on Zipf
The i th most frequent term has frequency proportional to 1/i Let this frequency be c/i. 500 , 000 Then i =1 c / i = 1. k Hk = i=11/ i. The k th Harmonic number is Thus c = 1/Hm , which is ~ 1/ln m = 1/ln(500k) ~ 1/13. So the i th most frequent term has frequency roughly 1/13i.
Postings analysis contd.
Expected number of occurrences of the i th most frequent term in a doc of length L is: Lc/i ~ L/13i ~ 76/i for L=1000. Let J = Lc ~ 76. Then the J most frequent terms are likely to occur in every document. Now imagine the term-document incidence matrix with rows sorted in decreasing order of term frequency:
3
Rows by decreasing frequency
n docs
J most frequent terms. J next most frequent terms. J next most frequent terms.
J-row blocks
In the i th of these J-row blocks, we have J rows each with n/i gaps of i each. Encoding a gap of i takes us 2log2 i +1 bits. So such a row uses space ~ (2n log2 i )/i bits. For the entire block, (2n J log2 i )/i bits, which in our case is ~ 1.5 x 108 (log2 i )/i bits. Sum this over i from 1 upto m/J = 500K/76~ 6500. (Since there are m/J blocks.)
n gaps of 1 each. n/2 gaps of 2 each. n/3 gaps of 3 each. m terms
etc.
Exercise
Work out the above sum and show it adds up to about 53 x 150 Mbits, which is about 1GByte. So weve taken 6GB of text and produced from it a 1GB index that can handle Boolean queries!
Caveats
This is not the entire space for our index:
does not account for dictionary storage next up; as we get further, well store even more stuff in the index.
Make sure you understand all the approximations in our probabilistic calculation.
Assumes Zipfs law applies to occurrence of terms in docs. All gaps for a term taken to be the same. Does not talk about query processing.
More practical caveat
codes are neat but in reality, machines have word boundaries 16, 32 bits etc
Compressing and manipulating at individual bitgranularity is overkill in practice Slows down architecture
Word-aligned compression
Simple example: fix a word-width (say 16 bits) Dedicate one bit to be a continuation bit c. If the gap fits within 15 bits, binary-encode it in the 15 available bits and set c=0. Else set c=1 and use additional words until you have enough bits for encoding the gap.
In practice, simpler word-aligned compression (see Scholer reference) better
4
Exercise
How would you adapt the space analysis for coded indexes to the scheme using continuation bits?
Exercise (harder)
How would you adapt the analysis for the case of positional indexes? Intermediate step: forget compression. Adapt the analysis to estimate the number of positional postings entries.
Dictionary and postings files
Term Doc # ambitious 2 be 2 brutus 1 brutus 2 capitol 1 caesar 1 caesar 2 did 1 enact 1 hath 2 I 1 i' 1 it 2 julius 1 killed 1 let 2 me 1 noble 2 so 2 the 1 the 2 told 2 you 2 was 1 was 2 with Freq 2 1 1 1 1 1 1 2 1 1 1 2 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1
Inverted index storage
Doc # 2 2 1 2 1 1 2 1 1 2 1 1 2 1 1 2 1 2 2 1 2 2 2 1 2 2 Freq 1 1 1 1 1 1 2 1 1 1 2 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1
Term N docs Tot Freq ambitious 1 1 be 1 1 brutus 2 2 capitol 1 1 caesar 2 3 did 1 1 enact 1 1 hath 1 1 I 1 2 i' 1 1 it 1 1 julius 1 1 killed 1 2 let 1 1 me 1 1 noble 1 1 so 1 1 the 2 2 told 1 1 you 1 1 was 2 2 with 1 1
Have estimated pointer storage Next up: Dictionary storage
Dictionary in main memory, postings on disk
This is common, especially for something like a search engine where high throughput is essential, but can also store most of it on disk with small, in-memory index
Tradeoffs between compression and query processing speed
Cascaded family of techniques
Usually in memory
Gap-encoded, on disk
How big is the lexicon V?
Grows (but more slowly) with corpus size Exercise: Can one Empirically okay model: derive this from m = kNb Zipfs Law? where b 0.5, k 30100; N = # tokens For instance TREC disks 1 and 2 (2 Gb; 750,000 newswire articles): ~ 500,000 terms V is decreased by case-folding, stemming Indexing all numbers could make it extremely large (so usually dont*) Spelling errors contribute a fair bit of size
Dictionary storage - first cut
Array of fixed-width entries
500,000 terms; 28 bytes/term = 14MB.
Terms a Freq. 999,712 Postings ptr.
aardvark 71 . zzzz . 99
Allows for fast binary 20 bytes search into dictionary
4 bytes each
5
Exercises
Is binary search really a good idea? What are the alternatives?
Fixed-width terms are wasteful
Most of the bytes in the Term column are wasted we allot 20 bytes for 1 letter terms.
And still cant handle supercalifragilisticexpialidocious.
Written English averages ~4.5 characters.
Exercise: Why is/isnt this the number to use for estimating the dictionary size? Explain this. Short words dominate token counts.
Average word in English: ~8 characters.
Compressing the term list
Store dictionary as a (long) string of characters:
Pointer to next word shows end of current word Hope to save up to 60% of dictionary space.
.systilesyzygeticsyzygialsyzygyszaibelyiteszczecinszomo.
Freq. 33 29 44 126 Postings ptr. Term ptr.
Total space for compressed list
4 bytes per term for Freq. 4 bytes per term for pointer to Postings. Now avg. 11 3 bytes per term pointer bytes/term, Avg. 8 bytes per term in term string not 20. 500K terms 9.5MB
Total string length = 500K x 8B = 4MB Pointers resolve 4M positions: log24M = 22bits = 3bytes
Binary search these pointers
Blocking
Store pointers to every kth on term string.
Example below: k=4.
Net
Where we used 3 bytes/pointer without blocking
3 x 4 = 12 bytes for k=4 pointers,
Need to store term lengths (1 extra byte)
.7systile9syzygetic8syzygial6syzygy11szaibelyite8szczecin9szomo.
now we use 3+4=7 bytes for 4 pointers.
Freq. 33 29 44 126 7
Postings ptr. Term ptr.
Shaved another ~0.5MB; can save more with larger k.
Save 9 bytes on 3 pointers. Lose 4 bytes on term lengths.
Why not go with larger k?
6
Exercise
Estimate the space usage (and savings compared to 9.5MB) with blocking, for block sizes of k = 4, 8 and 16.
Impact on search
Binary search down to 4-term block; Then linear search through terms in block. 8 documents: binary tree ave. = 2.6 compares Blocks of 4 (binary tree), ave. = 3 compares
3 5 7
2 4 6 8
1 1 5 6 2 7 3 8 4
= (1+22+43+4)/8
=(1+22+23+24+5)/8
Exercise
Estimate the impact on search performance (and slowdown compared to k=1) with blocking, for block sizes of k = 4, ...