hive set create table acl n int location

Hive set create table acl n int location

This preview shows page 58 - 60 out of 395 pages.

hive> set fs.s3.canned.acl=BucketOwnerFullControl; create table acl (n int) location 's3://acltestbucket/acl/'; insert overwrite table acl select count(n) from acl; The last two lines of the example create a table that is stored in Amazon S3 and write data to the table. To write files using canned ACLs in Pig From the Pig command prompt, set the fs.s3.canned.acl configuration option to the canned ACL you want to have the cluster set on files it writes to Amazon S3. To access the Pig command prompt connect to the master node using SSH, and type Pig at the Hadoop command prompt. For more information, see Connect to the Master Node Using SSH (p. 313) . The following example sets the fs.s3.canned.acl configuration option to BucketOwnerFullControl, which gives the owner of the Amazon S3 bucket complete control over the file. Note that the set command includes one space before the canned ACL name and contains no quotation marks. pig> set fs.s3.canned.acl BucketOwnerFullControl; store some data into 's3://acltestbucket/pig/acl'; To write files using canned ACLs in a custom JAR Set the fs.s3.canned.acl configuration option using Hadoop with the -D flag. This is shown in the example below. hadoop jar hadoop-examples.jar wordcount -Dfs.s3.canned.acl=BucketOwnerFullControl s3://mybucket/input s3://mybucket/output Compress the Output of your Cluster Topics Output Data Compression (p. 53) Intermediate Data Compression (p. 53) 52
Image of page 58
Amazon EMR Management Guide Plan and Configure Master Nodes Using the Snappy Library with Amazon EMR (p. 53) Output Data Compression This compresses the output of your Hadoop job. If you are using TextOutputFormat the result is a gzip'ed text file. If you are writing to SequenceFiles then the result is a SequenceFile which is compressed internally. This can be enabled by setting the configuration setting mapred.output.compress to true. If you are running a streaming job you can enable this by passing the streaming job these arguments. -jobconf mapred.output.compress=true You can also use a bootstrap action to automatically compress all job outputs. Here is how to do that with the Ruby client. --bootstrap-actions s3://elasticmapreduce/bootstrap-actions/configure-hadoop \ --args "-s,mapred.output.compress=true" Finally, if are writing a Custom Jar you can enable output compression with the following line when creating your job. FileOutputFormat.setCompressOutput(conf, true); Intermediate Data Compression If your job shuffles a significant amount data from the mappers to the reducers, you can see a performance improvement by enabling intermediate compression. Compress the map output and decompress it when it arrives on the core node. The configuration setting is mapred.compress.map.output. You can enable this similarly to output compression.
Image of page 59
Image of page 60

You've reached the end of your free preview.

Want to read all 395 pages?

  • Spring '12
  • LauraParker
  • Amazon Web Services, Amazon Elastic Compute Cloud

What students are saying

  • Left Quote Icon

    As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

    Student Picture

    Kiran Temple University Fox School of Business ‘17, Course Hero Intern

  • Left Quote Icon

    I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

    Student Picture

    Dana University of Pennsylvania ‘17, Course Hero Intern

  • Left Quote Icon

    The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

    Student Picture

    Jill Tulane University ‘16, Course Hero Intern

Stuck? We have tutors online 24/7 who can help you get unstuck.
A+ icon
Ask Expert Tutors You can ask You can ask ( soon) You can ask (will expire )
Answers in as fast as 15 minutes
A+ icon
Ask Expert Tutors