Update aws-sdk-java to 1.10.11 #234

lusid · 2015-08-17T21:29:44Z

Fixes #233

Amazon SDK v1.9.34 S3 functionality used by elasticsearch-cloud-aws no longer works after OpenJDK v1.8u60 due to JodaTime issues with timezone. More information here: aws/aws-sdk-java#444

dadoonet · 2015-08-17T22:09:46Z

See also elastic/elasticsearch#12859

lusid · 2015-08-18T10:54:31Z

Any idea how long it will take for a change like that to go through review and possible make it back to ES 1.5 (or 1.7 at least)?

dadoonet · 2015-08-18T11:51:36Z

For now I'm reopening your PR for this repo in case we plan to fix it for 1.7 and previous versions.

lusid · 2015-08-18T11:57:10Z

Thanks, I appreciate it. Considering this happened using the official ElasticSearch repository in DockerHub and was sparked by them making a modification to all of their base images as far as I can tell (in this case the java:8 repo), I would imagine there will be many others having this problem soon if they aren't already.

Follow up for #234. Release notes: http://aws.amazon.com/releasenotes/Java/6635368276326731 Closes #236. (cherry picked from commit 51d70f2) (cherry picked from commit bc56248) (cherry picked from commit 9781068) (cherry picked from commit a6d7787)

See #234 (cherry picked from commit 0a8fb47)

Follow up for #234. Release notes: http://aws.amazon.com/releasenotes/Java/6635368276326731 Closes #236. (cherry picked from commit 51d70f2)

See #234 (cherry picked from commit 0a8fb47)

Follow up for #234. Release notes: http://aws.amazon.com/releasenotes/Java/6635368276326731 Closes #236. (cherry picked from commit 51d70f2) (cherry picked from commit bc56248) (cherry picked from commit 9781068)

See #234 (cherry picked from commit 0a8fb47)

Follow up for #234. Release notes: http://aws.amazon.com/releasenotes/Java/6635368276326731 Closes #236. (cherry picked from commit 51d70f2) (cherry picked from commit bc56248)

See #234 (cherry picked from commit 0a8fb47)

Follow up for #234. Release notes: http://aws.amazon.com/releasenotes/Java/6635368276326731 Closes #236.

dadoonet · 2015-08-24T12:25:12Z

Anyone is up to run a test on a test cluster ?

bin/plugin install cloud-aws --url https://oss.sonatype.org/content/repositories/snapshots/org/elasticsearch/elasticsearch-cloud-aws/2.7.1-SNAPSHOT/elasticsearch-cloud-aws-2.7.1-20150824.122206-1.zip

The goal is to run elasticsearch 1.7 with this 2.7.1-SNAPSHOT version using either Java 1.7.0_85 or 1.8u60 (or >)

If you guys tries it, could you report here?

Thanks a lot for your help!

pvanderlinden · 2015-08-25T10:50:10Z

@dadoonet

I still have the same issue with the 2-7.1-SNAPSHOT cloud-aws plugin, running elasticsearch 1.7.1, Oracle java 1.8u60

[2015-08-25 10:46:30,036][WARN ][snapshots                ] [Anomalito] failed to create snapshot [abcde:sdfsdf]
org.elasticsearch.snapshots.SnapshotCreationException: [abcde:sdfsdf] failed to create snapshot
    at org.elasticsearch.repositories.blobstore.BlobStoreRepository.initializeSnapshot(BlobStoreRepository.java:260)
    at org.elasticsearch.snapshots.SnapshotsService.beginSnapshot(SnapshotsService.java:290)
    at org.elasticsearch.snapshots.SnapshotsService.access$600(SnapshotsService.java:92)
    at org.elasticsearch.snapshots.SnapshotsService$1$1.run(SnapshotsService.java:211)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Unable to upload object es-test-backup/snapshot-sdfsdf due to AmazonS3Exception: AWS authentication requires a valid Date or x-amz-date header (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 645B595151F0F941)
    at org.elasticsearch.cloud.aws.blobstore.DefaultS3OutputStream.upload(DefaultS3OutputStream.java:116)
    at org.elasticsearch.cloud.aws.blobstore.DefaultS3OutputStream.flush(DefaultS3OutputStream.java:90)
    at org.elasticsearch.cloud.aws.blobstore.S3OutputStream.flushBuffer(S3OutputStream.java:71)
    at org.elasticsearch.cloud.aws.blobstore.S3OutputStream.close(S3OutputStream.java:89)
    at org.elasticsearch.repositories.blobstore.BlobStoreRepository.initializeSnapshot(BlobStoreRepository.java:237)
    ... 6 more

dadoonet · 2015-08-25T12:52:20Z

@pvanderlinden can you tell me what you have in elasticsearch lib directory and in plugins/cloud-aws directory? Could you run a ls -l in both places?

Thanks!

Also, how did you install your JVM? I mean, download from oracle?

pvanderlinden · 2015-08-25T14:26:57Z

@dadoonet
The JVM is from an ubuntu PPA: https://launchpad.net/~webupd8team/+archive/ubuntu/java which does that automaticly for you.

The ls -l:

lib:
total 29208
-rw-r--r-- 1 root root   167735 Jul 29 09:56 antlr-runtime-3.5.jar
-rw-r--r-- 1 root root   448794 Jul 29 09:56 apache-log4j-extras-1.2.17.jar
-rw-r--r-- 1 root root    47359 Jul 29 09:56 asm-4.1.jar
-rw-r--r-- 1 root root    38090 Jul 29 09:56 asm-commons-4.1.jar
-rw-r--r-- 1 root root 14048559 Jul 29 09:56 elasticsearch-1.7.1.jar
-rw-r--r-- 1 root root  6998724 Jul 29 09:56 groovy-all-2.4.4.jar
-rw-r--r-- 1 root root   914597 Jul 29 09:56 jna-4.1.0.jar
-rw-r--r-- 1 root root   794991 Jul 29 09:56 jts-1.13.jar
-rw-r--r-- 1 root root   489884 Jul 29 09:56 log4j-1.2.17.jar
-rw-r--r-- 1 root root  1699875 Jul 29 09:56 lucene-analyzers-common-4.10.4.jar
-rw-r--r-- 1 root root  2563490 Jul 29 09:56 lucene-core-4.10.4.jar
-rw-r--r-- 1 root root    75491 Jul 29 09:56 lucene-expressions-4.10.4.jar
-rw-r--r-- 1 root root   107784 Jul 29 09:56 lucene-grouping-4.10.4.jar
-rw-r--r-- 1 root root   138282 Jul 29 09:56 lucene-highlighter-4.10.4.jar
-rw-r--r-- 1 root root    64021 Jul 29 09:56 lucene-join-4.10.4.jar
-rw-r--r-- 1 root root    36078 Jul 29 09:56 lucene-memory-4.10.4.jar
-rw-r--r-- 1 root root    97173 Jul 29 09:56 lucene-misc-4.10.4.jar
-rw-r--r-- 1 root root   213029 Jul 29 09:56 lucene-queries-4.10.4.jar
-rw-r--r-- 1 root root   391515 Jul 29 09:56 lucene-queryparser-4.10.4.jar
-rw-r--r-- 1 root root   119449 Jul 29 09:56 lucene-sandbox-4.10.4.jar
-rw-r--r-- 1 root root   126794 Jul 29 09:56 lucene-spatial-4.10.4.jar
-rw-r--r-- 1 root root   179011 Jul 29 09:56 lucene-suggest-4.10.4.jar
drwxr-xr-x 2 root root     4096 Aug 14 14:53 sigar
-rw-r--r-- 1 root root   102177 Jul 29 09:56 spatial4j-0.4.1.jar

plugins/cloud-aws/:
total 6456
-rw-r--r-- 1 root root  518808 Aug 25 14:23 aws-java-sdk-core-1.10.12.jar
-rw-r--r-- 1 root root 2166484 Aug 25 14:23 aws-java-sdk-ec2-1.10.12.jar
-rw-r--r-- 1 root root  258578 Aug 25 14:23 aws-java-sdk-kms-1.10.12.jar
-rw-r--r-- 1 root root  594380 Aug 25 14:23 aws-java-sdk-s3-1.10.12.jar
-rw-r--r-- 1 root root   58160 Aug 25 14:23 commons-codec-1.4.jar
-rw-r--r-- 1 root root   62050 Aug 25 14:23 commons-logging-1.1.3.jar
-rw-r--r-- 1 root root   47425 Aug 25 14:23 elasticsearch-cloud-aws-2.7.1-SNAPSHOT.jar
-rw-r--r-- 1 root root  590533 Aug 25 14:23 httpclient-4.3.5.jar
-rw-r--r-- 1 root root  282269 Aug 25 14:23 httpcore-4.3.2.jar
-rw-r--r-- 1 root root   39815 Aug 25 14:23 jackson-annotations-2.5.0.jar
-rw-r--r-- 1 root root  229998 Aug 25 14:23 jackson-core-2.5.3.jar
-rw-r--r-- 1 root root 1143162 Aug 25 14:23 jackson-databind-2.5.3.jar
-rw-r--r-- 1 root root  589289 Aug 25 14:23 joda-time-2.7.jar

dadoonet · 2015-08-25T14:43:15Z

@pvanderlinden Everything looks correct to me. I guess that you restarted the node, right?

I heard recently on a similar issue with openJDK 1.7.0.85.

It worked well when using redhead/centos openJDK 1.7.0.85 build 2.6.1.3.el6_6 but not when using openJDK 1.7.0.85 build 2.6.1.3.el6_7

Any chance here you have a similar issue?

pvanderlinden · 2015-08-25T14:57:23Z

@dadoonet I restarted the node indeed.

Not sure what your question is? What I had before I upgraded: Oracle Java 1.8u45 with cloud-aws on version 2.5.0, that still works fine in an old test instance. On an other test instance I completely removed java, and all downloaded packages for java, reinstalled, but it is still failing with the same error.

dadoonet · 2015-08-25T20:27:03Z

Did you run at some point ES 1.5 + CloudAWS 2.5.0 but with Latest Java Oracle version on the same machine?
Was it working?

I'm trying to understand where the issue is coming from. JVM? Cloud plugin? AWS SDK? ...

pvanderlinden · 2015-08-26T10:12:08Z

@dadoonet I tried a couple of things on a test machine and other installations
Working for me: 1.8u45 with ES 1.5/1.7/1.7.1 and plugin 2.5.0/2.7.1-snapshot
Not working for me: 1.8.u60 with ES 1.7/1.7.1 with plugin 2.7.1-snaphot/2.5.0
I did not try ES 1.5 with 1.8u45

dadoonet · 2015-08-26T10:16:55Z

Sorry you wrote:

Working for me: 1.8u45 with ES 1.5/1.7/1.7.1 and plugin 2.5.0/2.7.1-snapshot

And then

I did not try ES 1.5 with 1.8u45

You meant you did not try ES 1.5 with 1.8u60, right?

Whatever, I saw this comment today: aws/aws-sdk-java#444 (comment)

Might be the same concern... Will investigate more.

pvanderlinden · 2015-08-26T10:26:31Z

I meant that I didn't try ES 1.5 with 1.8u60 indeed.

dadoonet · 2015-08-26T18:01:46Z

Ok I know what is wrong. I'll update the SNAPSHOT and report again here. Thanks for your help so far!

dadoonet · 2015-08-26T18:10:29Z

Could you try:

bin/plugin install cloud-aws --url https://oss.sonatype.org/content/repositories/snapshots/org/elasticsearch/elasticsearch-cloud-aws/2.7.1-SNAPSHOT/elasticsearch-cloud-aws-2.7.1-20150826.180934-2.zip

Thanks!

pvanderlinden · 2015-08-26T19:43:20Z

The new snapshot works fine, create, restore and list snapshot all works again, Thanks.

dadoonet · 2015-08-26T19:44:10Z

\o/ Thanks again for your help! I really appreciate. Planning to release soonish now.

Follow up for discussion #234 We need to pull in Joda 2.8.2 and don't rely of the version declared by elasticsearch which is currently 2.7. It will fix issues seen with recent Java versions. See also aws/aws-sdk-java#444 Closes #239. (cherry picked from commit 0db449a)

Follow up for discussion #234 We need to pull in Joda 2.8.2 and don't rely of the version declared by elasticsearch which is currently 2.7. It will fix issues seen with recent Java versions. See also aws/aws-sdk-java#444 Closes #239.

pvanderlinden · 2015-08-26T19:52:24Z

Thanks for fixing. This should fix my last problem with the ELK stack hopefully.

peter-svensson · 2015-08-27T11:37:52Z

Will there be a release 2.7.1 with this fix anytime soon?

dadoonet · 2015-08-27T12:37:22Z

@pliljenberg Definitely yes! Very very soon!

dadoonet · 2015-08-27T14:24:29Z

@pliljenberg @pvanderlinden @lusid 2.7.1 and 2.6.1 have been released.

dbaggott · 2015-09-23T20:35:05Z

@dadoonet, any chance of getting 2.7.1 into the maven repository? As best as I can see, only 2.7.0 is available. Thanks!

dadoonet · 2015-09-23T20:42:22Z

@dbaggott weird! Sounds like it was partially released. Indeed, artifacts are not in maven repo.
Will fix it hopefully tomorrow. Thanks for reporting!

dbaggott · 2015-09-23T20:44:10Z

@dadoonet, thank you!

dbaggott · 2015-09-23T20:46:23Z

ps you probably already noted this but all of the fixed versions are missing: 2.4.3, 2.5.2, etc

dadoonet · 2015-09-23T20:47:30Z

No I didn't. Need to fix that.

dadoonet · 2015-09-24T11:48:45Z

@dbaggott I looked at it.

2.4.2 is the latest for 1.4
2.5.1 is the latest for 1.5
2.6.1 is the latest for 1.6: this one is missing
2.7.1 is the latest for 1.7: this one is missing

I just pushed the last 2 versions. It will need some minutes I guess to be synchronized on maven central.

Thanks again for raising that!

dbaggott · 2015-09-24T13:17:19Z

Fantastic, thank you. I can confirm that both 2.7.1 and 2.6.1 are now available (despite not showing up in the UI yet).

smiller171 · 2015-09-29T21:12:01Z

Has this fix been backported to ES 1.5?

dadoonet · 2015-09-29T21:58:03Z

No. You should update to elasticsearch 1.7 and update AWS plugin accordingly.

dadoonet · 2015-09-29T21:58:52Z

I meant it has been backported but this branch has not been released since.

Follow up for elastic#234. Release notes: http://aws.amazon.com/releasenotes/Java/6635368276326731 Closes elastic#236. (cherry picked from commit 51d70f2) (cherry picked from commit bc56248) (cherry picked from commit 9781068) (cherry picked from commit a6d7787)

EdgeCaseBerg · 2017-08-30T18:39:56Z

@smiller171
I assume you already figured out a solution, but just in case someone else comes along and is really trying to get something to work. I got the snapshot/repository stuff working by doing the following:

Install 2.5.2 snapshot

bin/plugin install cloud-aws --url https://oss.sonatype.org/content/repositories/snapshots/org/elasticsearch/elasticsearch-cloud-aws/2.5.2-SNAPSHOT/elasticsearch-cloud-aws-2.5.2-20150822.225952-55.zip

Download some jar files:
Joda 2.8.2
AWS SDK Core 1.10.15
AWS SDK Core 1.10.15
mv the old versions of those jars out of the plugins/cloud-aws directory and move those new jars in
Restart ES

It worked for me doing this. But I only have to do this for a local environment running Java 8 I have (since we're running Java 7 where our actual cluster is) so I don't think I'd really recommending downloading a snapshot and changing out jars in the plugin directory. It'd probably be a little safer to replace the other AWS jars as well, but like I said, I didn't because on my local environment I didn't need to test any of the other features of the cloud-aws plugin

Sorry for resurrecting an old thread after the solution for people running 1.5 is to migrate to a different version was given by the maintainer!

Update aws-sdk-java to 1.10.11

0a8fb47

lusid closed this Aug 18, 2015

dadoonet reopened this Aug 18, 2015

dadoonet added 2.4.3 2.5.2 2.6.1 2.7.1 update labels Aug 24, 2015

dadoonet mentioned this pull request Aug 24, 2015

Update aws-sdk-java to 1.10.12 #236

Closed

dadoonet merged commit 0a8fb47 into elastic:master Aug 24, 2015

dadoonet pushed a commit that referenced this pull request Aug 24, 2015

Update aws-sdk-java to 1.10.11

a24031e

See #234 (cherry picked from commit 0a8fb47)

dadoonet added a commit that referenced this pull request Aug 24, 2015

Update aws-sdk-java to 1.10.12

bc56248

Follow up for #234. Release notes: http://aws.amazon.com/releasenotes/Java/6635368276326731 Closes #236. (cherry picked from commit 51d70f2)

dadoonet pushed a commit that referenced this pull request Aug 24, 2015

Update aws-sdk-java to 1.10.11

4e86bbf

See #234 (cherry picked from commit 0a8fb47)

dadoonet pushed a commit that referenced this pull request Aug 24, 2015

Update aws-sdk-java to 1.10.11

b7d64af

See #234 (cherry picked from commit 0a8fb47)

dadoonet added a commit that referenced this pull request Aug 24, 2015

Update aws-sdk-java to 1.10.12

9781068

Follow up for #234. Release notes: http://aws.amazon.com/releasenotes/Java/6635368276326731 Closes #236. (cherry picked from commit 51d70f2) (cherry picked from commit bc56248)

dadoonet pushed a commit that referenced this pull request Aug 24, 2015

Update aws-sdk-java to 1.10.11

5bf9cad

See #234 (cherry picked from commit 0a8fb47)

dadoonet added a commit that referenced this pull request Aug 24, 2015

Update aws-sdk-java to 1.10.12

51d70f2

Follow up for #234. Release notes: http://aws.amazon.com/releasenotes/Java/6635368276326731 Closes #236.

dadoonet mentioned this pull request Aug 24, 2015

Update AWS SDK to 1.10.10 elastic/elasticsearch#12859

Closed

dadoonet mentioned this pull request Aug 26, 2015

SDK not working on upcoming Java 1.8u60 aws/aws-sdk-java#444

Closed

dadoonet mentioned this pull request Aug 26, 2015

Force inclusion of Joda 2.8.2 #239

Closed

dadoonet mentioned this pull request Aug 27, 2015

Fix AWS API issue when running Java v1.8u60 #240

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update aws-sdk-java to 1.10.11 #234

Update aws-sdk-java to 1.10.11 #234

lusid commented Aug 17, 2015

dadoonet commented Aug 17, 2015

lusid commented Aug 18, 2015

dadoonet commented Aug 18, 2015

lusid commented Aug 18, 2015

dadoonet commented Aug 24, 2015

pvanderlinden commented Aug 25, 2015

dadoonet commented Aug 25, 2015

pvanderlinden commented Aug 25, 2015

dadoonet commented Aug 25, 2015

pvanderlinden commented Aug 25, 2015

dadoonet commented Aug 25, 2015

pvanderlinden commented Aug 26, 2015

dadoonet commented Aug 26, 2015

pvanderlinden commented Aug 26, 2015

dadoonet commented Aug 26, 2015

dadoonet commented Aug 26, 2015

pvanderlinden commented Aug 26, 2015

dadoonet commented Aug 26, 2015

pvanderlinden commented Aug 26, 2015

peter-svensson commented Aug 27, 2015

dadoonet commented Aug 27, 2015

dadoonet commented Aug 27, 2015

dbaggott commented Sep 23, 2015

dadoonet commented Sep 23, 2015

dbaggott commented Sep 23, 2015

dbaggott commented Sep 23, 2015

dadoonet commented Sep 23, 2015

dadoonet commented Sep 24, 2015

dbaggott commented Sep 24, 2015

smiller171 commented Sep 29, 2015

dadoonet commented Sep 29, 2015

dadoonet commented Sep 29, 2015

EdgeCaseBerg commented Aug 30, 2017

Update aws-sdk-java to 1.10.11 #234

Update aws-sdk-java to 1.10.11 #234

Conversation

lusid commented Aug 17, 2015

dadoonet commented Aug 17, 2015

lusid commented Aug 18, 2015

dadoonet commented Aug 18, 2015

lusid commented Aug 18, 2015

dadoonet commented Aug 24, 2015

pvanderlinden commented Aug 25, 2015

dadoonet commented Aug 25, 2015

pvanderlinden commented Aug 25, 2015

dadoonet commented Aug 25, 2015

pvanderlinden commented Aug 25, 2015

dadoonet commented Aug 25, 2015

pvanderlinden commented Aug 26, 2015

dadoonet commented Aug 26, 2015

pvanderlinden commented Aug 26, 2015

dadoonet commented Aug 26, 2015

dadoonet commented Aug 26, 2015

pvanderlinden commented Aug 26, 2015

dadoonet commented Aug 26, 2015

pvanderlinden commented Aug 26, 2015

peter-svensson commented Aug 27, 2015

dadoonet commented Aug 27, 2015

dadoonet commented Aug 27, 2015

dbaggott commented Sep 23, 2015

dadoonet commented Sep 23, 2015

dbaggott commented Sep 23, 2015

dbaggott commented Sep 23, 2015

dadoonet commented Sep 23, 2015

dadoonet commented Sep 24, 2015

dbaggott commented Sep 24, 2015

smiller171 commented Sep 29, 2015

dadoonet commented Sep 29, 2015

dadoonet commented Sep 29, 2015

EdgeCaseBerg commented Aug 30, 2017