Advanced CPS216: Database Systems (Data-intensive Computing Systems) Introduction to MapReduce and Hadoop
Shivnath Babu
Word Count over a Given Set of Web Pages
see bob throw see bob throw see spot run 1 1 1 1 1 1 bob run see spot throw 1 1 2 1 1
see spot run
Can we do word count in parallel?
The MapReduce Framework (pioneered by Google)
Automatic Parallel Execution in MapReduce (Google)
Handles failures automatically, e.g., restarts tasks if a node fails; runs multiples copies of the same task to avoid a slow task slowing down the whole job
MapReduce in Hadoop (1)
MapReduce in Hadoop (2)
MapReduce in Hadoop (3)
Data Flow in a MapReduce Program in Hadoop
InputFormat Map function Sorting Partitioner & Merging Combiner Shuffling Merging Reduce function OutputFormat
1:many
Lifecycle of a MapReduce Job
Map function
Reduce function
Run this program as a MapReduce job
Lifecycle of a MapReduce Job
Map function
Reduce function
Run this program as a MapReduce job
Lifecycle of a MapReduce Job
Time
Input Splits
Map Wave 1
Map Wave 2
Reduce Wave 1
Reduce Wave 2
How are the number of splits, number of map and reduce tasks, memory allocation to tasks, etc., determined?
Job Configuration Parameters
190+ parameters in Hadoop Set manually or defaults are used
How to sort data using Hadoop?
Let us look at a complete example MapReduce program in Hadoop