class05-index-construction

class05-index-construction - Index construction How do we...

Info iconThis preview shows pages 1–5. Sign up to view the full content.

View Full Document Right Arrow Icon
Index construction How do we construct an index? What strategies can we use with limited main memory? Hardware basics Many design decisions in IR are based on the characteristics of hardware We begin by reviewing some hardware basics . ..
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Hardware basics Access to data in memory is much faster than access to data on disk (~10x) Disk seeks are “idle” time: No data is transferred from disk while the disk head is being positioned To optimize transfer time from disk to memory: one large chunk is faster than many small chunks Disk I/O is block-based: Reading and writing of entire blocks (as opposed to smaller chunks). Block sizes: 8KB to 256KB Hardware basics Servers used in IR systems now typically have several GB of main memory, sometimes tens of GB Available disk space is several orders of magnitude larger: TBs or 100s of GBs Fault tolerance is very expensive: It’s much cheaper to use many regular machines rather than one fault tolerant machine
Background image of page 2
Some stats !()*!*+ ( 2 *( 3 2 4 56 78 ( 9 6=6> ?( 3 > 4 56 7@ ( 56 D ( 75 + 6=65 ?( 3 56 7@ ( (JK& -; (JK& -; RCV1: Our test collection today Shakespeare’s collected works are not large enough for demonstrating many of the points in this course As an example for applying scalable index construction algorithms, we’ll use the Reuters RCV1 collection English newswire articles in 1995-96
Background image of page 3

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
! !()*!*+ ! . 3445444 ! 6 +789 : $0;"2& <"% /0- =44 ! > D445444 ! +789 : EB$"& <"% $0;"2 F ! +789 : EB$"& <"% $0;"2 D9J !
Background image of page 4
Image of page 5
This is the end of the preview. Sign up to access the rest of the document.

Page1 / 16

class05-index-construction - Index construction How do we...

This preview shows document pages 1 - 5. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online