The trade off of course is that if something goes wrong youll have fewer tools

The trade off of course is that if something goes

This preview shows page 142 - 144 out of 395 pages.

The trade off, of course, is that if something goes wrong, you'll have fewer tools available to investigate the issue. Default Log Files By default, each cluster writes log files on the master node. These are written to the /mnt/var/log/ directory. You can access them by using SSH to connect to the master node as described in Connect to the Master Node Using SSH (p. 313) . Because these logs exist on the master node, when the node terminates—either because the cluster was shut down or because an error occurred—these log files are no longer available. You do not need to enable anything to have log files written on the master node. This is the default behavior of Amazon EMR and Hadoop. A cluster generates several types of log files, including: Step logs — These logs are generated by the Amazon EMR service and contain information about the cluster and the results of each step. The log files are stored in /mnt/var/log/hadoop/steps/ directory on the master node. Each step logs its results in a separate numbered subdirectory: / mnt/var/log/hadoop/steps/s- stepId1 / for the first step, /mnt/var/log/hadoop/steps/ s- stepId2 / , for the second step, and so on. The 13-character step identifiers (e.g. stepId1, stepId2) are unique to a cluster. Hadoop and YARN component logs — The logs for components associated with both Apache YARN and MapReduce, for example, are contained in separate folders in /mnt/var/log . The log file locations for the Hadoop components under /mnt/var/log are as follows: hadoop-hdfs, hadoop- mapreduce, hadoop-httpfs, and hadoop-yarn. The hadoop-state-pusher directory is for the output of the Hadoop state pusher process. Bootstrap action logs — If your job uses bootstrap actions, the results of those actions are logged. The log files are stored in /mnt/var/log/bootstrap-actions/ on the master node. Each bootstrap action logs its results in a separate numbered subdirectory: /mnt/var/log/bootstrap-actions/1/ for the first bootstrap action, /mnt/var/log/bootstrap-actions/2/ , for the second bootstrap action, and so on. 136
Image of page 142
Amazon EMR Management Guide Archive Log Files to Amazon S3 Instance state logs — These logs provide information about the CPU, memory state, and garbage collector threads of the node. The log files are stored in /mnt/var/log/instance-state/ on the master node. Archive Log Files to Amazon S3 Note You cannot currently use log aggregation to Amazon S3 with the yarn logs utility. You can configure a cluster to periodically archive the log files stored on the master node to Amazon S3. This ensures that the log files are available after the cluster terminates, whether this is through normal shut down or due to an error. Amazon EMR archives the log files to Amazon S3 at 5 minute intervals. To have the log files archived to Amazon S3, you must enable this feature when you launch the cluster.
Image of page 143
Image of page 144

You've reached the end of your free preview.

Want to read all 395 pages?

  • Spring '12
  • LauraParker
  • Amazon Web Services, Amazon Elastic Compute Cloud

What students are saying

  • Left Quote Icon

    As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

    Student Picture

    Kiran Temple University Fox School of Business ‘17, Course Hero Intern

  • Left Quote Icon

    I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

    Student Picture

    Dana University of Pennsylvania ‘17, Course Hero Intern

  • Left Quote Icon

    The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

    Student Picture

    Jill Tulane University ‘16, Course Hero Intern

Ask Expert Tutors You can ask You can ask ( soon) You can ask (will expire )
Answers in as fast as 15 minutes