01_Intro_MR_cluster_access

01_Intro_MR_cluster_access - Introduction to MapReduce...

This preview shows page 1. Sign up to view the full content.

This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: Introduction to MapReduce Programming & Local Hadoop Cluster Accesses Instructions Rozemary Scarlat August 31, 2011 DataClow in a MR Program (K1, V1) (K2, V2) (K2, List<V2>) (K3, V3) We have temperature readings for the years 1901 2001 and we want to compute the maximum for each year In our temperature data set, each line looks like this: 0067011990999991950051507004...9999999N9+00001+99999999999... 0043011990999991950051512004...9999999N9+00221+99999999999... We know that characters 16 19 represent the year, characters 88 92 represent the temperature and character 93 represents the quality code (0, 0067011990999991950051507004...9999999N9+00001+99999999999...) (106, 0043011990999991950051512004...9999999N9+00221+99999999999...) Implementation in MapReduce (0, 0067011... ) Selection+ Projection (1950, 22) 0 1950, 22 -11 Aggregation (MAX) (1950, 22) Mapper Reducer Main Beyond MaxTemperature What if we want to get the average temperature for a year? What if you are only interested in the temperature in Durham? (Assume the station ID at Durham is 212) Local Hadoop Cluster The master node is hadoop21.cs.duke.edu The slave nodes are hadoop[22,24-36].cs.duke.edu Online jobtracker address*: http://hadoop21.cs.duke.edu:50030/jobtracker.jsp Online HDFS health*: http://hadoop21.cs.duke.edu:50070/dfshealth.jsp * Accesible only from within the CS trusted network. Solution: 1. ssh to any node and then use lynx. 2. build "ssh -D port" connection to any node, set proxy in your browser Now, let's see how to compile and run a MapReduce job on the local cluster You can Cind the detailed instructions at the course website: http://www.cs.duke.edu/courses/fall10/cps216/TA_material/ cluster_instructions Mapper (old API) Reducer (old API) Main (old API) ...
View Full Document

{[ snackBarMessage ]}

Ask a homework question - tutors are online