Related Topics View Log Files p 283 Use a Custom Bootstrap Action to Copy an

Related topics view log files p 283 use a custom

This preview shows page 109 - 111 out of 395 pages.

Related Topics View Log Files (p. 283) Use a Custom Bootstrap Action to Copy an Object from Amazon S3 to Each Node You can use a bootstrap action to copy objects from Amazon S3 to each node in a cluster before your applications are installed. The AWS CLI is installed on each node of a cluster, so your bootstrap action can call AWS CLI commands. The following example demonstrates a simple bootstrap action script that copies a file, myfile.jar , from Amazon S3 to a local folder, /mnt1/myfolder , on each cluster node. The script is saved to Amazon S3 with the file name copymyfile.sh with the following contents. 103
Image of page 109
Amazon EMR Management Guide Configure Cluster Hardware and Networking #!/bin/bash aws s3 cp s3://mybucket/myfilefolder/myfile.jar /mnt1/myfolder When you launch the cluster, you specify the script. The following AWS CLI example demonstrates this: aws emr create-cluster --name "Test cluster" --release-label emr-5.28.1 \ --use-default-roles --ec2-attributes KeyName=myKey \ --applications Name=Hive Name=Pig \ --instance-count 3 --instance-type m5.xlarge \ --bootstrap-actions Path="s3://mybucket/myscriptfolder/copymyfile.sh" Configure Cluster Hardware and Networking An important consideration when you create an EMR cluster is how you configure Amazon EC2 instances and network options. EC2 instances in an EMR cluster are organized into node types . There are three: the master node , the core node , and task nodes . Each node type performs a set of roles defined by the distributed applications that you install on the cluster. During a Hadoop MapReduce or Spark job, for example, components on core and task nodes process data, transfer output to Amazon S3 or HDFS, and provide status metadata back to the master node. With a single-node cluster, all components run on the master node. The collection of EC2 instances that host each node type is called either an instance fleet or a uniform instance group . The instance fleets or uniform instance groups configuration is a choice you make when you create a cluster. It applies to all node types, and it can't be changed later. When you create a cluster, you make choices that ultimately determine the performance profile of your cluster. This chapter covers those options in detail, and then ties them all together with best practices and guidelines. Note The instance fleets configuration is available only in Amazon EMR release versions 4.8.0 and later, excluding 5.0.0 and 5.0.3. Topics Understanding Master, Core, and Task Nodes (p. 104) Configure EC2 Instances (p. 106) Configure Networking (p. 111) Create a Cluster with Instance Fleets or Uniform Instance Groups (p. 120) Cluster Configuration Guidelines and Best Practices (p. 131) Understanding Master, Core, and Task Nodes Use this section to understand how Amazon EMR uses each of these node types and as a foundation for cluster capacity planning.
Image of page 110
Image of page 111

You've reached the end of your free preview.

Want to read all 395 pages?

  • Spring '12
  • LauraParker
  • Amazon Web Services, Amazon Elastic Compute Cloud

What students are saying

  • Left Quote Icon

    As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

    Student Picture

    Kiran Temple University Fox School of Business ‘17, Course Hero Intern

  • Left Quote Icon

    I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

    Student Picture

    Dana University of Pennsylvania ‘17, Course Hero Intern

  • Left Quote Icon

    The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

    Student Picture

    Jill Tulane University ‘16, Course Hero Intern

Stuck? We have tutors online 24/7 who can help you get unstuck.
A+ icon
Ask Expert Tutors You can ask You can ask ( soon) You can ask (will expire )
Answers in as fast as 15 minutes
A+ icon
Ask Expert Tutors