SEARCH WEB FOR A PLANET: THE GOOGLE CLUSTER ARCHITECTURE
AMENABLE TO EXTENSIVE PARALLELIZATION, GOOGLES WEB SEARCH
APPLICATION LETS DIFFERENT QUERIES RUN ON DIFFERENT PROCESSORS AND, BY PARTITIONING THE OVERALL INDEX, ALSO LETS A SINGLE QUERY USE MULTIPLE PROCESSORS. TO HANDLE THIS WORKLOAD, GOOGLES ARCHITECTURE FEATURES CLUSTERS OF MORE THAN 15,000 COMMODITYCLASS PCS WITH FAULT-TOLERANT SOFTWARE. THIS ARCHITECTURE ACHIEVES SUPERIOR PERFORMANCE AT A FRACTION OF THE COST OF A SYSTEM BUILT FROM FEWER, BUT MORE EXPENSIVE, HIGH-END SERVERS.
Luiz Andr Barroso Jeffrey Dean Urs Hlzle Google
Few Web services require as much computation per request as search engines. On average, a single query on Google reads hundreds of megabytes of data and consumes tens of billions of CPU cycles. Supporting a peak request stream of thousands of queries per second requires an infrastructure comparable in size to that of the largest supercomputer installations. Combining more than 15,000 commodity-class PCs with fault-tolerant software creates a solution that is more cost-effective than a comparable system built out of a smaller number of high-end servers. Here we present the architecture of the Google cluster, and discuss the most important factors that inuence its design: energy efciency and price-performance ratio. Energy efciency is key at our scale of operation, as power consumption and cooling issues become signicant operational factors, taxing the lim-
its of available data center power densities. Our application affords easy parallelization: Different queries can run on different processors, and the overall index is partitioned so that a single query can use multiple processors. Consequently, peak processor performance is less important than its price/ performance. As such, Google is an example of a throughput-oriented workload, and should benefit from processor architectures that offer more on-chip parallelism, such as simultaneous multithreading or on-chip multiprocessors.
Google architecture overview
Googles software architecture arises from two basic insights. First, we provide reliability in software rather than in server-class hardware, so we can use commodity PCs to build a high-end computing cluster at a low-end
22
Published by the IEEE Computer Society
0272-1732/03/$17.00 2003 IEEE
price. Second, we tailor the design for best aggregate request throughput, not peak server response time, since we can manage response times by parallelizing individual requests. We believe that the best price/performance tradeoff for our applications comes from fashioning a reliable computing infrastructure from clusters of unreliable commodity PCs. We provide rel...
googlecluster-ieee