376 Amazon EMR Management Guide VPC Errors If you use Amazon EMR release

376 amazon emr management guide vpc errors if you use

This preview shows page 382 - 385 out of 395 pages.

376
Image of page 382
Amazon EMR Management Guide VPC Errors If you use Amazon EMR release versions earlier than 5.0, you can use the following command in HiveQL to pre-cache the results of an Amazon S3 list operation locally on the cluster: set hive.optimize.s3.query=true; Use static partitions where possible. In some versions of Hive and Amazon EMR, it is possible that using ALTER TABLES will fail because the table is stored in a different location than expected by Hive. The solution is to add or update following in /home/hadoop/conf/core-site.xml : <property> <name>fs.s3n.endpoint</name> <value>s3.amazonaws.com</value> </property> VPC Errors The following errors are common to VPC configuration in Amazon EMR. Topics Invalid Subnet Configuration (p. 377) Missing DHCP Options Set (p. 377) Permissions Errors (p. 378) Errors That Result in START_FAILED (p. 378) Cluster Terminated with errors and NameNode Fails to Start (p. 379) Invalid Subnet Configuration On the Cluster Details page, in the Status field, you see an error similar to the following: The subnet configuration was invalid: Cannot find route to InternetGateway in main RouteTable rtb-id for vpc vpc-id . To solve this problem, you must create an Internet Gateway and attach it to your VPC. For more information, see Adding an Internet Gateway to Your VPC . Alternatively, verify that you have configured your VPC with Enable DNS resolution and Enable DNS hostname support enabled. For more information, see Using DNS with Your VPC . Missing DHCP Options Set You see a step failure in the cluster system log (syslog) with an error similar to the following: ERROR org.apache.hadoop.security.UserGroupInformation (main): PriviledgedActionException as:hadoop (auth:SIMPLE) cause:java.io.IOException: org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application with id ' application_id ' doesn't exist in RM. or ERROR org.apache.hadoop.streaming.StreamJob (main): Error Launching job : org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application with id ' application_id ' doesn't exist in RM. 377
Image of page 383
Amazon EMR Management Guide VPC Errors To solve this problem, you must configure a VPC that includes a DHCP Options Set whose parameters are set to the following values: Note If you use the AWS GovCloud (US-West) region, set domain-name to us-gov- west-1.compute.internal instead of the value used in the following example. domain-name = ec2.internal Use ec2.internal if your region is US East (N. Virginia). For other regions, use region-name .compute.internal . For example in us-west-2, use domain-name = us- west-2.compute.internal . domain-name-servers = AmazonProvidedDNS For more information, see DHCP Options Sets . Permissions Errors A failure in the stderr log for a step indicates that an Amazon S3 resource does not have the appropriate permissions. This is a 403 error and the error looks like: Exception in thread "main" com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: REQUEST_ID If the ActionOnFailure is set to TERMINATE_JOB_FLOW
Image of page 384
Image of page 385

You've reached the end of your free preview.

Want to read all 395 pages?

  • Spring '12
  • LauraParker
  • Amazon Web Services, Amazon Elastic Compute Cloud

What students are saying

  • Left Quote Icon

    As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

    Student Picture

    Kiran Temple University Fox School of Business ‘17, Course Hero Intern

  • Left Quote Icon

    I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

    Student Picture

    Dana University of Pennsylvania ‘17, Course Hero Intern

  • Left Quote Icon

    The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

    Student Picture

    Jill Tulane University ‘16, Course Hero Intern

Stuck? We have tutors online 24/7 who can help you get unstuck.
A+ icon
Ask Expert Tutors You can ask You can ask ( soon) You can ask (will expire )
Answers in as fast as 15 minutes
A+ icon
Ask Expert Tutors