10.1.1.109.4049.pdf - The Anatomy of a Large-Scale Hypertextual Web Search Engine Sergey Brin and Lawrence Page Isergey pagel cs.stanford.edu Computer

The Anatomy of a Large-Scale Hypertextual Web SearchEngineSergey Brin and Lawrence PageIsergey, [email protected]Computer Science Department, Stanford University, Stanford, CA 94305AbstractIn this paper, we present Google, a prototype of a large-scale search engine which makes heavy use of thestructure present in hypertext.Google is designed to crawl and index the Web efficiently and produce much moresatisfying search results than existing systems.The prototype with a full text and hyperlink database of at least 24million pages is available at http:[email protected]/To engineer a search engine is a challenging task.Search engines index tens to hundreds of millions of webpages involving a comparable number of distinct terms.They answer tens of n-tillions of queries every day.Despite the importance of large-scale search engines on the web, very little academic research has been done onthem.Furthermore, due to rapid advance in technology and web proliferation, creating a web search engine todayis very different from three years ago.This paper provides an in-depth description of our large-scale web searchengine -- the first such detailed public description we know of to date.Apart from the problems of scaling traditional search techniques to data of this magnitude, there are newtechnicalchallenges involved with using the additional information present in hypertext to produce better search results.This paper addresses this question of how to build a practical large-scale system which can exploit the additionalinfon-nation present in hypertext.Also we look at the problem of how to effectively deal with uncontrolledhypertext collections where anyone can publish anything they want.Keywords:World Wide Web, Search Engines, Information Retrieval, PageRank, Google

1.Introduction(Note:There are two versions of this paper - - a longerfull version and a shorter printed version.The full version is available on the web andtheconference CD-ROM.)The web creates new challenges for information retrieval.The amount of information on the web is growing rapidly, as well as the number ofnew users inexperienced in the art of web research.People are likely to surf the web using its link graph, often starting with high qualityhuman maintained indices such as Yahoo! or with search engines.Human maintained lists cover popular topics effectively but are subjective,expensive to build and maintain, slow to improve, and cannot cover all esoteric topics.Automated search engines that rely on keywordmatching usually return too many low quality matches.To make matters worse, some advertisers attempt to gain people's attention by takingmeasures meant to mislead automated search engines.We have built a large-scale search engine which addresses many of the problems ofexisting systems.It makes especially heavy use of the additional structure present in hypertext to provide much higher quality search results.