Exception in S3: AWS authentication requires a valid Date or x-amz-date header

[ferozed]

I am getting the following exception when doing an S3 directory list operation:

com.amazonaws.services.s3.model.AmazonS3Exception: AWS authentication requires a valid Date or x-amz-date header (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 5EF25AEC935F76D5) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)

I know this is an old issue ( #484, #444 ). I made sure that I am doing exactly what is recommended there.

This is my setup:

• I have amazon sdk version 1.10.6
• joda-time 2.8.1
• I use java JDK8 1.8.0_112-b15
• we use the shade plugin to package all dependencies into one jar so that we do not get conflicts.

I am making these calls from a Map Reduce job that runs in an on-premise cluster. I was under the impression that upgrading joda-time to a version > 2.7.0 resolved the incompatibilities between joda-time and the JDK8.

please help me resolve this issue.

[varunnvs92]

Is it possible that another version of joda-time is being pulled by another dependency?
Can you try printing the version of joda-time using this:
System.out.println(new DateTime().getClass().getProtectionDomain().getCodeSource());

[ferozed]

Yeah did that already as per the other issues. It is correct.

2016-12-09 11:23:51,324 INFO [main] [com.mycompany.lib] (file:/var/lib/hadoop-yarn/cache/yarn/nm-local-dir/filecache/21/process-map-reduce-1.0.765.0-hotfix_joda-incompat-test.09a9076.jar <no signer certificates>)

Note that as I have explained, I bundle joda-time, aws-java-sdk and most dependencies in my jar using maven-shade-plugin.

[ferozed]

Any takers? This is blocking us.

[shorea]

Can you try upgrading to the latest version of the SDK (1.11.66)? This issues has been patched in the SDK code so it wouldn't matter what version of joda time you actually use at runtime.

[ferozed]

We cant upgrade the sdk that easily. The issue is that our code runs on a on-premise cloudera hadoop cluster, that ships with 1.10.6 version. If I use the newer one it will be incompatible with the one that ships in the cloudera distribution.

[shorea]

Is it possible to upgrade within the 1.10 family? They are backwards compatible with each other.

I was looking at the release notes and we apparently fixed this in 1.10.24
https://aws.amazon.com/releasenotes/Java/1476077219718774

[pjsg]

In my case, it turned out that jruby-cloudera's version of joda-time was being used. The latest version of jruby-cloudera is 1.0.0 which includes joda-time 1.6.2 (ARGH!!!) This is an explicit dependency of snakeyaml 1.9

I've filed a ticket with snakeyaml to fix their dependency issue: https://bitbucket.org/asomov/snakeyaml/issues/357/old-version-of-joda-time-16-is-included-in

However, the version of snakeyaml included in the jruby-cloudera jar is also very old.

My google-fu is weak today -- I can't find where jruby-cloudera comes from... Most pom.xml files indicate that it just contains jzlib -- but that isn't the case. https://www.google.com/webhp?ie=UTF-8#q=%22jruby-cloudera%22+%22snakeyaml%22

[asomov]

SnakeYAML does not have any dependency. It has a few test dependencies.
Why is it important ?

[shorea]

If you update to the latest of 1.10 or 1.11 of the SDK, it doesn't matter which version of joda time you pull in.

[pjsg]

I hadn't spotted that it was a test dependency. Maybe jruby-cloudera just pulls in an old version of joda-time.

[pjsg]

I'm running sdk 1.10.49 (as far as I can tell) -- I'll upgrade to 1.10.77 (which seems to be current).

[shorea]

Hmm strange. Thought it was fixed with 1.10.24. If upgrading to 1.10.77 doesn't work let me know.

[varunnvs92]

Are you still having the issue?

[pjsg]

I got into dependency hell and gave up.

[The backstory is that I was fixing some unrelated code, when I saw this exception go past. It turns out to only happen in one particular case while running integration tests. I wasn't feeling sufficiently motivated to track it down. I just fixed the issue that I had, and moved on -- leaving our tech debt for some other engineer.....]

[varunnvs92]

Closing the issue for now. When you have more details, feel free to re-open the ticket. Thank you.