cis6930fa11_Map-Reduce_Online

cis6930fa11_Map-Reduce_Online - MapReduce Online Tyson...

Info iconThis preview shows pages 1–10. Sign up to view the full content.

View Full Document Right Arrow Icon
MapReduce Online Tyson Condie, Neil Conway, Peter Alvaro, Joseph Hellerstein, Khaled Elmeleegy, Russell Sears Neeraj Ganapathy
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Outline Hadoop Architecture Pipelined MapReduce Online Aggregation Continuous MapReduce Jobs Performance Analysis Future Work
Background image of page 2
Hadoop Architecture Hadoop MapReduce User defined Map, Reduce functions Job Tracker accepts jobs from clients Job is divided into smaller tasks and assigned to slave nodes Hadoop Distributed File System(HDFS) Files are stored in fixed size blocks(64MB default) Stores input to Map and output from Reduce Output of Map and Reduce tasks are written to local file before it can be consumed Simple fault tolerance mechanism
Background image of page 3

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Dataflow in Hadoop Submit job schedule map map reduce reduce
Background image of page 4
Dataflow – Map phase HDFS Block 1 Block 2 map map reduce reduce Read Input File
Background image of page 5

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Dataflow – Commit phase map map reduce reduce Local FS Local FS Finished Finished + Location
Background image of page 6
Dataflow – Shuffle phase map map reduce reduce Local FS Local FS HTTP GET
Background image of page 7

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Dataflow – Sort, Reduce phase reduce reduce HDFS Write Final Answer
Background image of page 8
Pipelined MapReduce Advantages of Pipelining Online aggregation is possible Support for continuous queries Better performance
Background image of page 9

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Image of page 10
This is the end of the preview. Sign up to access the rest of the document.

This note was uploaded on 11/09/2011 for the course CIS 6930 taught by Professor Staff during the Fall '08 term at University of Florida.

Page1 / 32

cis6930fa11_Map-Reduce_Online - MapReduce Online Tyson...

This preview shows document pages 1 - 10. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online