If you reconfigure Spark applications in a running cluster the application

If you reconfigure spark applications in a running

This preview shows page 287 - 290 out of 395 pages.

If you reconfigure Spark applications in a running cluster, the application history will be not available through the Spark history server UI. For each AWS account, the number of active Spark history server UIs cannot exceed 50. You can access Spark history server UI from the console in the US East (N.Virgina and Ohio), US West (N.California and Oregon), Canada (Central), EU (Frankfurt, Ireland, and London), and Asia Pacific (Mumbai, Seoul, Singapore, Sydney, and Tokyo) Regions. Access Application History through Spark History Server UI On the Application history tab or the cluster Summary page for your cluster in the Amazon EMR console, choose the Spark history server UI link. The Spark history server UI opens in a new browser tab. This web interface displays the same information as the open source Spark HistoryServer UI if you set up a web proxy through an SSH connection. For more information, see Monitoring and Instrumentation . You can view YARN container logs through the links on the Spark history server UI. Note To access YARN container logs from the Spark history server UI, you must enable logging to Amazon S3 for your cluster. If logging is not enabled, the links to YARN container logs will not work. 281
Image of page 287
Amazon EMR Management Guide View Application History View a Summary of Application History With Amazon EMR versions 5.8.0 and later, you can view a summary of the application history from the Application history tab in the Amazon EMR console. Amazon EMR keeps the summary of application history for seven days after an application has completed. The following sequence demonstrates a drill-down through a Spark or YARN application into job details using the Application history on a cluster details page. To view cluster details, from the Clusters list, select a cluster Name . To view information about YARN container logs, you must enable logging for your cluster. For more information, see Configure Cluster Logging and Debugging . For Spark application history, the information provided in the summary table is only a subset of the information available through Spark history server UI. In the following Application history tab example, two rows are expanded to show the diagnostic summaries for two different Spark applications, and an Application ID is selected to view further application detail. On the Jobs tab of YARN application details, the description of Job 0 is selected to see Job 0 details. On the Job 0 details page, information about individual job stages is expanded, and then the Description for Stage 1 is selected to see Stage 1 details. 282
Image of page 288
Amazon EMR Management Guide View Log Files On the Stage 1 details page, key metrics for stage tasks and executors can be seen, and task and executor logs can be viewed using links.
Image of page 289
Image of page 290

You've reached the end of your free preview.

Want to read all 395 pages?

  • Spring '12
  • LauraParker
  • Amazon Web Services, Amazon Elastic Compute Cloud

What students are saying

  • Left Quote Icon

    As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

    Student Picture

    Kiran Temple University Fox School of Business ‘17, Course Hero Intern

  • Left Quote Icon

    I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

    Student Picture

    Dana University of Pennsylvania ‘17, Course Hero Intern

  • Left Quote Icon

    The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

    Student Picture

    Jill Tulane University ‘16, Course Hero Intern

Ask Expert Tutors You can ask You can ask ( soon) You can ask (will expire )
Answers in as fast as 15 minutes