As loc wban datechararray temp count grunt dataflt

AS (loc, wban, date:chararray, temp, count);
grunt> data_flt = FILTER data_raw BY date != ’YEARMODA’;
grunt> data = FOREACH data_flt GENERATE (long)loc, (int)wban,
date, (float)temp, (int)count;
grunt> temps = FOREACH data GENERATE ((temp-32.0)*5.0/9.0);
grunt> temps_group = GROUP temps ALL;
grunt> max_temp = FOREACH temps_group GENERATE MAX(temps);
grunt> DUMP max_temp;
(43.27777862548828)
16

Also the mean daily temperatures were obtained from averaging a variable number of
measurements: the amount is given in the 5
th
column, variable count. You might want
to filter all mean values obtained with less than –say– 5 measurements out. This is left
as an exercise to the reader.
5.4 Some extra Pig commands
Some relational operators
FILTER Use it to work with tuples or rows of data
FOREACH Use it to work with columns of data
GROUP Use it to group data in a single relation
ORDER Sort a relation based on one or more fields
...
Some built-in functions
AVG Calculate the average of numeric values in a single-column bag
COUNT Calculate the number of tuples in a bag
MAX/MIN Calculate the maximum/minimum value in a single-column bag
SUM Calculate the sum of values in a single-column bag
...
17

Hands-on block 6
Extras
6.1 Installing your own Hadoop
The Hadoop community has its main online presence in:
Although you can download the latest source code and release tarballs from that location,
we strongly suggest you to use the more production-ready Cloudera distribution:
Cloudera provides ready to use Hadoop Linux packages for several distributions, as well as
a Hadoop Installer for configuring your own Hadoop cluster, and also a VMWare appliance
preconfigured with Hadoop, Hue, HBase and more.
18

Appendix A
Additional Information
Hadoop Homepage
Internet:
Cloudera Hadoop Distribution
Internet:
Documentation
Tutorial:
mapred_tutorial.html
Hadoop API:
Pig:
Recommended books
Hadoop: The
Definitive Guide
Tom White, O’Reilly Media, 2010 (2nd Ed.)
Hadoop in action Chuck Lam, Manning, 2011
19