The following AWS CLI command for Amazon S3 replaces a file saved to a local

The following aws cli command for amazon s3 replaces

This preview shows page 34 - 36 out of 395 pages.

The following AWS CLI command for Amazon S3 replaces a file saved to a local machine called SharedNotebook.ipynb for an EMR notebook with the name MyNotebook , an ID of e-12A3BCDEFJHIJKLMNO45PQRST , and created with MyBucket/MyNotebooksFolder specified in Amazon S3. For information about using the Amazon S3 console to copy and replace files, see Uploading, Downloading, and Managing Objects in the Amazon Simple Storage Service Console User Guide . aws s3 cp SharedNotebook.ipynb s3://MyBucket/ MyNotebooksFolder/-12A3BCDEFJHIJKLMNO45PQRST/MyNotebook.ipynb Monitoring Spark User and Job Activity EMR Notebooks allows you to configure user impersonation on a Spark cluster. This feature helps you track job activity initiated from within the notebook editor. In addition, EMR Notebooks has a built-in Jupyter Notebook widget that lets you to view Spark job details alongside query output in the notebook editor. The widget is available by default and requires no special configuration. However, to view the history servers, your client must be configured to view Amazon EMR web interfaces that are hosted on the master node. 28
Image of page 34
Amazon EMR Management Guide Setting Up Spark User Impersonation Setting Up Spark User Impersonation By default, Spark jobs that users submit using the notebook editor appear to originate from an indistinct livy user identity. You can configure user impersonation for the cluster so that these jobs are associated with the IAM user identity that ran the code instead. HDFS user directories on the master node are created for each user identity that runs code in the notebook. For example, if user NbUser1 runs code from the notebook editor, you can connect to the master node and see that hadoop fs -ls /user shows the directory /user/user_NbUser1 . You enable this feature by setting properties in the core-site and livy-conf configuration classifications. This feature is not available by default when you have Amazon EMR create a cluster along with a notebook. For more information about using configuration classifications to customize applications, see Configuring Applications in the Amazon EMR Release Guide . Use the following configuration classifications and values to enable user impersonation for EMR Notebooks: [ { "Classification": "core-site", "Properties": { "hadoop.proxyuser.livy.groups": "*", "hadoop.proxyuser.livy.hosts": "*" } }, { "Classification": "livy-conf", "Properties": { "livy.impersonation.enabled": "true" } } ] Using the Spark Job Monitoring Widget When you run code in the notebook editor that execute Spark jobs on the EMR cluster, the output includes a Jupyter Notebook widget for Spark job monitoring. The widget provides job details and useful links to the Spark history server page and the Hadoop job history page, along with convenient links to job logs in Amazon S3 for any failed jobs.
Image of page 35
Image of page 36

You've reached the end of your free preview.

Want to read all 395 pages?

  • Spring '12
  • LauraParker
  • Amazon Web Services, Amazon Elastic Compute Cloud

What students are saying

  • Left Quote Icon

    As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

    Student Picture

    Kiran Temple University Fox School of Business ‘17, Course Hero Intern

  • Left Quote Icon

    I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

    Student Picture

    Dana University of Pennsylvania ‘17, Course Hero Intern

  • Left Quote Icon

    The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

    Student Picture

    Jill Tulane University ‘16, Course Hero Intern

Ask Expert Tutors You can ask You can ask ( soon) You can ask (will expire )
Answers in as fast as 15 minutes