The following example submits a bash script AddHDFSUserssh to a cluster that

The following example submits a bash script

This preview shows page 237 - 239 out of 395 pages.

The following example submits a bash script AddHDFSUsers.sh to a cluster that already exists, referencing its cluster ID. The script is saved to Amazon S3. aws emr add-steps --cluster-id ClusterID \ --steps Type=CUSTOM_JAR,Name=CustomJAR,ActionOnFailure=CONTINUE,\ Jar=s3:// MyRegion .elasticmapreduce/libs/script-runner/script-runner.jar,Args=[" s3:// MyBucketPath/AddHDFSUsers.sh "] The following example demonstrates the contents of the AddHDFSUsers.sh script. #!/bin/bash # AddHDFSUsers.sh script # Initialize an array of user names from AD, or Linux users created manually on the cluster ADUSERS=(" lijuan " " marymajor " " richardroe " " myusername ") # For each user listed, create an HDFS user directory # and change ownership to the user for username in ${ADUSERS[@]}; do hdfs dfs -mkdir /user/$username hdfs dfs -chown $username:$username /user/$username done 231
Image of page 237
Amazon EMR Management Guide Use Kerberos Authentication Enabling GSSAPI for SSH For Kerberos-authenticated users to connect to the master node using SSH, the SSH service must have GSSAPI authentication enabled. To enable GSSAPI, run the following commands from the master node command line or use a step to run it as a script. After reconfiguring SSH, you must restart the service. sudo sed -i 's/^.*GSSAPIAuthentication.*$/GSSAPIAuthentication yes/' /etc/ssh/sshd_config sudo sed -i 's/^.*GSSAPICleanupCredentials.*$/GSSAPICleanupCredentials yes/' /etc/ssh/ sshd_config sudo /etc/init.d/sshd restart Using SSH to Connect to Kerberized Clusters This section demonstrates the steps for a Kerberos-authenticated user to connect to the master node of an EMR cluster. Each computer that is used for an SSH connection must have SSH client and Kerberos client applications installed. Linux computers most likely include these by default. For example, OpenSSH is installed on most Linux, Unix, and macOS operating systems. You can check for an SSH client by typing ssh at the command line. If your computer does not recognize the command, install an SSH client to connect to the master node. The OpenSSH project provides a free implementation of the full suite of SSH tools. For more information, see the OpenSSH website. Windows users can use applications such as PuTTY as an SSH client. For more information about SSH connections, see Connect to the Cluster (p. 313) . SSH uses GSSAPI for authenticating Kerberos clients, and you must enable GSSAPI authentication for the SSH service on the cluster master node. For more information, see Enabling GSSAPI for SSH (p. 232) . SSH clients must also use GSSAPI. In the following examples, for MasterPublicDNS use the value that appears for Master public DNS on the Summary tab of the cluster details pane—for example, ec2-11-222-33-44.compute-1.amazonaws.com . Prerequisite for krb5.conf (Non Active Directory) When using a configuration without Active Directory integration, in addition to the SSH client and Kerberos client applications, each client computer must have a copy of the /etc/krb5.conf file that matches the /etc/krb5.conf file on the cluster master node.
Image of page 238
Image of page 239

You've reached the end of your free preview.

Want to read all 395 pages?

  • Spring '12
  • LauraParker
  • Amazon Web Services, Amazon Elastic Compute Cloud

What students are saying

  • Left Quote Icon

    As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

    Student Picture

    Kiran Temple University Fox School of Business ‘17, Course Hero Intern

  • Left Quote Icon

    I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

    Student Picture

    Dana University of Pennsylvania ‘17, Course Hero Intern

  • Left Quote Icon

    The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

    Student Picture

    Jill Tulane University ‘16, Course Hero Intern

Ask Expert Tutors You can ask You can ask ( soon) You can ask (will expire )
Answers in as fast as 15 minutes