Log Files on the Master Node Every cluster publishes logs files to the

Log files on the master node every cluster publishes

This preview shows page 359 - 361 out of 395 pages.

Log Files on the Master Node Every cluster publishes logs files to the /mnt/var/log/ directory on the master node. These log files are only available while the cluster is running. Log Files Archived to Amazon S3 If you launch the cluster and specify an Amazon S3 log path, the cluster copies the log files stored in / mnt/var/log/ on the master node to Amazon S3 in 5-minute intervals. This ensures that you have access to the log files even after the cluster is terminated. Because the files are archived in 5-minute intervals, the last few minutes of an suddenly terminated cluster may not be available. Tools to Monitor Cluster Performance Amazon EMR provides several tools to monitor the performance of your cluster. Hadoop Web Interfaces Every cluster publishes a set of web interfaces on the master node that contain information about the cluster. You can access these web pages by using an SSH tunnel to connect them on the master node. For more information, see View Web Interfaces Hosted on Amazon EMR Clusters (p. 318) . CloudWatch Metrics Every cluster reports metrics to CloudWatch. CloudWatch is a web service that tracks metrics, and which you can use to set alarms on those metrics. For more information, see Monitor Metrics with CloudWatch (p. 296) . Viewing and Restarting Amazon EMR and Application Processes (Daemons) When you troubleshoot a cluster, you may want to list running processes. You may also find it useful to stop or restart processes in some circumstances—for example, after you change a configuration or notice a problem with a particular process after you analyze log files and error messages. There are two types of processes that run on a cluster: Amazon EMR processes (for example, instance- controller and Log Pusher), and processes associated with the applications installed on the cluster (for example, hadoop-hdfs-namenode, and hadoop-yarn-resourcemanager). 353
Image of page 359
Amazon EMR Management Guide Viewing Running Processes To work with processes directly on a cluster, you connect to the master node. For more information, see Connect to the Cluster (p. 313) . Viewing Running Processes If you are using Amazon EMR version 4.x or later, application releases are packaged using a system based on Apache Bigtop, so these application processes are configured via .conf scripts under the upstart init system. Amazon EMR processes, on the other hand, are configured using SysV (init.d scripts) which is backwards compatible with upstart. To view a list of running Amazon EMR processes Type the following command (without the $ , which indicates the Linux command prompt): $ ls /etc/init.d/ The command returns a list of running Amazon EMR processes similar to the following example: acpid cloud-init-local instance-controller ntpd To view a list of processes associated with application releases Type the following command: $ ls /etc/init/ The command returns a list of running application processes similar to the following example: control-alt-delete.conf hadoop-yarn-resourcemanager.conf hive- metastore.conf Restarting Processes After you determine which processes are running, you can stop and then restart them if necessary. How
Image of page 360
Image of page 361

You've reached the end of your free preview.

Want to read all 395 pages?

  • Spring '12
  • LauraParker
  • Amazon Web Services, Amazon Elastic Compute Cloud

What students are saying

  • Left Quote Icon

    As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

    Student Picture

    Kiran Temple University Fox School of Business ‘17, Course Hero Intern

  • Left Quote Icon

    I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

    Student Picture

    Dana University of Pennsylvania ‘17, Course Hero Intern

  • Left Quote Icon

    The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

    Student Picture

    Jill Tulane University ‘16, Course Hero Intern

Ask Expert Tutors You can ask You can ask ( soon) You can ask (will expire )
Answers in as fast as 15 minutes