lecture5-compression-handout-6-per

Blocksof4binarytreeavg 1222324583compares 27

Info iconThis preview shows page 1. Sign up to view the full content.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: the
space
 for
the
dic)onary
and
pos)ngs
   Basic
Boolean
index
only
   No
study
of
posi)onal
indexes,
etc.
   We
will
consider
compression
schemes
 DICTIONARY
COMPRESSION
 17
 18
 3 Introduc)on to Informa)on Retrieval Sec. 5.2 Sec. 5.2 Introduc)on to Informa)on Retrieval Why
compress
the
dic)onary?
 Dic)onary
storage
‐
first
cut
   Search
begins
with
the
dic)onary
   We
want
to
keep
it
in
memory
   Memory
footprint
compe))on
with
other
 applica)ons
   Embedded/mobile
devices
may
have
very
li]le
 memory
   Even
if
the
dic)onary
isn’t
in
memory,
we
want
it
to
 be
small
for
a
fast
search
startup
)me
   So,
compressing
the
dic)onary
is
important
   Array
of
fixed‐width
entries
   ~400,000
terms;
28
bytes/term
=
11.2
MB.
 19
 Introduc)on to Informa)on Retrieval Sec. 5.2 Dictionary search structure 20 bytes 4 bytes each 20
 Sec. 5.2 Introduc)on to Informa)on Retrieval Compressing
the
term
list:

 Dic)onary‐as‐a‐String
 Fixed‐width
terms
are
wasteful
 Store dictionary as a (long) string of characters:   Most
of
the
bytes
in
the
Term
column
are
wasted
–
 we
allot
20
bytes
for
1
le]er
terms.
   Pointer to next word shows end of current word Hope to save up to 60% of dictionary space.       And
we
s)ll
can’t
handle
supercalifragilis)cexpialidocious or
 hydrochlorofluorocarbons. ….systilesyzygeticsyzygialsyzygyszaibelyiteszczecinszomo….   Wri]en
English
averages
~4.5
characters/word.
   Exercise:
Why
is/isn’t
this
the
number
to
use
for
es)ma)ng
 the
dic)onary
size?
 Total string length = 400K x 8B = 3.2MB   Ave.
dic)onary
word
in
English:
~8
characters
 Pointers resolve 3.2M positions: log23.2M = 22bits = 3bytes   How
do
we
use
~8
characters
per
dic)onary
term?
   Short
words
dominate
token
counts
but
not
type
 average.
 21
 Introduc)on to Informa)on Retrieval Sec. 5.2 Space
for
dic)onary
as
a
string
           22
 Sec. 5.2 Introduc)on to Informa)on Retrieval Blo...
View Full Document

This document was uploaded on 02/26/2014.

Ask a homework question - tutors are online