02_Beyond_mapper_reducer

02_Beyond_mapper_reducer - Beyond Mapper and Reducer...

Info iconThis preview shows pages 1–6. Sign up to view the full content.

View Full Document Right Arrow Icon
Beyond Mapper and Reducer Rozemary Scarlat September 13, 2011 Partitioner, Combiner, Hadoop Parameters and
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Data flow with multiple reducers
Background image of page 2
Partitioner § the map tasks partition their output, each creating one partition for each reduce task § many keys per partition, but all records for a key are in a single partition § default partitioner: HashPartitioner - hashes a record’s key to determine which partition the record belongs in § another partitioner: TotalOrderPartitioner – creates a total order by reading split points from an externally generated source
Background image of page 3

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
§ The partitioning can be controlled by a user-defined partitioning function: § Don’t forget to set the partitioner class: job.setPartitionerClass(OurPartitioner. class ); § Useful information about partitioners: - Hadoop book – Total Sort (pg. 237); Multiple Outputs (pg. 244); - http://chasebradford.wordpress.com/2010/12/12/reusable-total-order-sor - http://philippeadjiman.com/blog/2009/12/20/hadoop-tutorial-series-issue / (Note: uses the old API!)
Background image of page 4
Partitioner example
Background image of page 5

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Image of page 6
This is the end of the preview. Sign up to access the rest of the document.

This document was uploaded on 01/17/2012.

Page1 / 11

02_Beyond_mapper_reducer - Beyond Mapper and Reducer...

This preview shows document pages 1 - 6. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online