You can also view Amazon CloudWatch metrics for your EMRFS metadata in the

You can also view amazon cloudwatch metrics for your

This preview shows page 73 - 75 out of 395 pages.

You can also view Amazon CloudWatch metrics for your EMRFS metadata in the DynamoDB console where you can see the number of throttled read and write requests. If you do have a non-zero value for throttled requests, your application may potentially benefit from increasing allocated throughput capacity for read or write operations. You may also realize a performance benefit if you see that your operations are approaching the maximum allocated throughput capacity in reads or writes for an extended period of time. Throughput characteristics for notable EMRFS operations The default for read and write operations is 400 and 100 throughput capacity units, respectively. The following performance characteristics give you an idea of what throughput is required for certain operations. These tests were performed using a single-node m3.large cluster. All operations were single threaded. Performance differs greatly based on particular application characteristics and it may take experimentation to optimize file system operations. Operation Average read-per- second Average write-per-second create (object) 26.79 6.70 delete (object) 10.79 10.79 delete (directory containing 1000 objects) 21.79 338.40 getFileStatus (object) 34.70 0 getFileStatus (directory) 19.96 0 listStatus (directory containing 1 object) 43.31 0 listStatus (directory containing 10 objects) 44.34 0 listStatus (directory containing 100 objects) 84.44 0 listStatus (directory containing 1,000 objects) 308.81 0 listStatus (directory containing 10,000 objects) 416.05 0 listStatus (directory containing 100,000 objects) 823.56 0 67
Image of page 73
Amazon EMR Management Guide Consistent View Operation Average read-per- second Average write-per-second listStatus (directory containing 1M objects) 882.36 0 mkdir (continuous for 120 seconds) 24.18 4.03 mkdir 12.59 0 rename (object) 19.53 4.88 rename (directory containing 1000 objects) 23.22 339.34 To submit a step that purges old data from your metadata store Users may wish to remove particular entries in the DynamoDB-based metadata. This can help reduce storage costs associated with the table. Users have the ability to manually or programmatically purge particular entries by using the EMRFS CLI delete subcommand. However, if you delete entries from the metadata, EMRFS no longer makes any checks for consistency. Programmatically purging after the completion of a job can be done by submitting a final step to your cluster, which executes a command on the EMRFS CLI. For instance, type the following command to submit a step to your cluster to delete all entries older than two days. aws emr add-steps --cluster-id j-2AL4XXXXXX5T9 --steps Name="emrfsCLI",Jar="command- runner.jar",Args=["emrfs","delete","--time","2","--time-unit","days"] { "StepIds": [ " s-B12345678902 " ] } Use the StepId value returned to check the logs for the result of the operation. Configure Consistency Notifications for CloudWatch and Amazon SQS You can enable CloudWatch metrics and Amazon SQS messages in EMRFS for Amazon S3 eventual consistency issues.
Image of page 74
Image of page 75

You've reached the end of your free preview.

Want to read all 395 pages?

  • Spring '12
  • LauraParker
  • Amazon Web Services, Amazon Elastic Compute Cloud

What students are saying

  • Left Quote Icon

    As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

    Student Picture

    Kiran Temple University Fox School of Business ‘17, Course Hero Intern

  • Left Quote Icon

    I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

    Student Picture

    Dana University of Pennsylvania ‘17, Course Hero Intern

  • Left Quote Icon

    The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

    Student Picture

    Jill Tulane University ‘16, Course Hero Intern

Stuck? We have tutors online 24/7 who can help you get unstuck.
A+ icon
Ask Expert Tutors You can ask You can ask ( soon) You can ask (will expire )
Answers in as fast as 15 minutes
A+ icon
Ask Expert Tutors