Amazon Complaining about Missing Date Header When Creating/Fetching S3 Repository

[lusid]

I'm currently using v2.5.1 of this plugin successfully on one of my servers (running inside the official ES Docker image) to create and snapshot to an S3 repository. I recently started a new instance of ES in a new Docker container on the same server with the same exact configuration with the intention of restoring a snapshot, but noticed I could not create repositories or fetch listings of snapshots that exist in those repositories from the new instance. On the original instance, everything still works as expected, but now I'm scared to shut down that Docker container since it is working correctly at the moment. As a side note, I did this exact same process just a couple days ago with success, so I'm not sure what has changed since Wednesday.

This is the error I'm getting:

{
   "error": "RemoteTransportException[[server][inet[/192.168.1.1:9300]][cluster:admin/repository/put]]; nested: RepositoryVerificationException[[my_s3_repository] path [my_s3_path] is not accessible on master node]; nested: IOException[Unable to upload object my_s3_path/tests-ooBzS8deRtWxFeS6OuG9SQ-master]; nested: AmazonS3Exception[AWS authentication requires a valid Date or x-amz-date header (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: C810733A376C4D79)]; ",
   "status": 500
}

Any idea what would be causing this? Thanks in advance!

[daviddyball]

I'm also experiencing this issue using Docker+Elasticsearch 1.4 container and 2.4.1 of the elasticsearch-cloud-aws plugin.

curl -XPUT 'http://127.0.0.1:9200/_snapshot/backups' -d '{
    "type":"s3",
    "settings": {
        "bucket":"mybackups",
        "region":"eu-west-1",
        "protocol":"https"
    }
}'

{
  "status": 500,
  "error": "RepositoryVerificationException[[backups] path  is not accessible on master node]; nested: IOException[Unable to upload object tests-LL5FIlrPS86_InDfnNEzvw-master]; nested: AmazonS3Exception[AWS authentication requires a valid Date or x-amz-date header (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 622050F6EEA5C8C2)]; "
}

[lusid]

Seems to be related to some change in the Docker base images: aws/aws-sdk-java#484

[lusid]

It seems the required change is to use Amazon SDK v1.10.11 for any version of OpenJDK 1.8u60 and above (in the case of the base Docker images, 1.8u66 is the new base version used for 1.8).

I'm not exactly sure how to go about testing this or if there is something else that needs to be done to get this change into all of the versions, but I tried modifying the pom.xml file in v1.5 branch and run a "mvn clean install", but get the following error:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.2:testCompile (default-testCompile) on project elasticsearch-cloud-aws: Compilation failure
[ERROR] /Users/marcmelvin/Development/SmartProcure/elasticsearch-cloud-aws/src/test/java/org/elasticsearch/cloud/aws/AmazonS3Wrapper.java:[41,7] error: AmazonS3Wrapper is not abstract and does not override abstract method getBucketReplicationConfiguration(GetBucketReplicationConfigurationRequest) in AmazonS3

[luismfonseca]

@lusid Thats because that class no longer implements all the methods present in the AmazonS3.

I've actually implemented them, packaged a new version with the latest Amazon SDK dependency but it still gives the same error.

Upon inspecting the generated zip, I saw that it still includes joda-time 1.7.0. This comes from elasticsearch-parent project [pom].

Forcing to use the latest joda-time will issue an runtime error on ES:

1) Error injecting constructor, java.lang.NoClassDefFoundError: org/joda/time/ReadableInstant
  at org.elasticsearch.repositories.s3.S3Repository.<init>(Unknown Source)
  while locating org.elasticsearch.repositories.s3.S3Repository
  while locating org.elasticsearch.repositories.Repository

1 error
    at org.elasticsearch.common.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:344)
    at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:178)
    at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
    at org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:131)
    at org.elasticsearch.common.inject.ModulesBuilder.createChildInjector(ModulesBuilder.java:69)
    at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:407)
    at org.elasticsearch.repositories.RepositoriesService.clusterChanged(RepositoriesService.java:302)
    at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:480)
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:196)
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:162)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoClassDefFoundError: org/joda/time/ReadableInstant
    at com.amazonaws.services.s3.internal.ServiceUtils.<clinit>(ServiceUtils.java:67)
    at com.amazonaws.services.s3.internal.S3Signer.sign(S3Signer.java:168)
(...)

I think Elasticsearch is not loading the joda-time present in the plugin, so I think Elasticsearch's joda-time needs to be updated to use the latest version.

[lusid]

dadoonet linked here on my pull request where they updated the aws-sdk in ES 5 days ago to 1.10.10: elastic/elasticsearch#12859

I guess we are just waiting then at this point.

[TheDeveloper]

Seeing this too.

I'd say this is pretty urgent given people wanting to recover into a new cluster can't because they can't create the repository to recover from.

[TheDeveloper]

Requests to recover shards from S3 are also failing:

[2015-08-21 12:20:04,337][WARN ][cluster.action.shard     ] [Carter Ghazikhanian] [anon-67827][2] received shard failed for [anon-67827][2], node[BjiX6YqASgqqLeOOhRCMwA], [P], restoring[s3-hourly:2015-08-21-09-32-51-0000], s[INITIALIZING], unassigned_info[[reason=ALLOCATION_FAILED], at[2015-08-21T12:20:03.832Z], details[shard failure [failed recovery][IndexShardGatewayRecoveryException[[anon-67827][2] failed recovery]; nested: IndexShardRestoreFailedException[[anon-67827][2] restore failed]; nested: IndexShardRestoreFailedException[[anon-67827][2] failed to restore snapshot [2015-08-21-09-32-51-0000]]; nested: AmazonS3Exception[AWS authentication requires a valid Date or x-amz-date header (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: E26]0A722C546D490)]; ]], indexUUID [I7ndH_d9RWiymLlZou3cTQ], reason [master [Carter Ghazikhanian][l6dioplgRjaOrL3tr8-6aw][elasticsearch-1-12][inet[/10.0.150.57:9300]] marked shard as initializing, but shard is marked as failed, resend shard failure]

Our workaround for this was to downgrade to a JVM version before 1.8u60 (thanks to @lusid for the tip). In our case we went back to 1.7.0_80.

The cause appears to be a compatibility issue between JodaTime and later versions of java 8. See http://stackoverflow.com/questions/32058431/aws-java-sdk-aws-authentication-requires-a-valid-date-or-x-amz-date-header

[lusid]

My workaround was similar. Downgrading to 1.8u45 solves it as well.

[rodriguezsergio]

Any chance this fix will be applied to java version "1.8.0_73" ?

Running
elasticsearch 1.4.5-1
and plugin 2.4.2

[dadoonet]

@rodriguezsergio 2.4.3 won't be released so you have two options IMO:

• Upgrade elasticsearch to 1.7 and aws plugin to 2.7.1 (best option IMO)
• build yourself branch es-1.4 and install the plugin from target dir

Also, you could think of upgrading to 2.x which is even better but requires more work probably on your end.

[rodriguezsergio]

@dadoonet Yeah, we're trying to play catch-up with you Elastic guys. Stop releasing so much 😛. Will upgrade to 1.7. Thanks.