Put task to an S3 bucket fails even though the actual upload succeeded

[jyriok]

Hello,

i use S3 compatible storage versioned for store my bbr backup.

so i use this :

- name: AF-storage
  type: s3
  source:
    bucket: ee.cloud-backup
    endpoint: 10.x.x.10:8080
    disable_ssl: true
    use_v2_signing: true
    versioned_file: backup/cf-backup.tar
    access_key_id: SOMESECRETSOMESECRET
    secret_access_key: SOMESECRETSOMESECRET

and job :

  - put: AF-storage
    params:
      file: backup/cf-backup.tar

and when my pipeline is done and concourse push the file on S3, i've this error message :

3.48 MB / 3.48 MB [=====================================] 100.00 % 16.59 MB/s 0serror running command: object versioning not enabled

but my file is rightly stored and versioned on my bucket !

what's i'm doing wrong ? :o

thanks !

[cappyzawa]

@jyriok Hello.
According to out/command.go, it seems that the error occurs when versioned_file is non empty and VersionId is empty.

VersionId seems to be included in the metadata although it is not related to the pipeline's yml.

Could you please check resource screen (http:///teams//pipelines//resources/AF-storage) after put?

[jyriok]

thanks for your help !

i've do a new try and take screen :

as u see, version ID is set and they are the same between S3 storage & s3 resource :(

[jyriok]

[cappyzawa]

@jyriok Sorry for the late reply.
hmmm...

i use S3 compatible storage versioned for store my bbr backup.

It also reproduced in my environment.
I am using S3 compatible storage, too.

There is something I want you to try.

$ aws s3api list-object-versions --bucket <your bucket> --endpoint-url <your strage url>

If cli response An error occurred (MethodNotAllowed) when calling the ListObjectVersions operation: The specified method is not allowed against this resource., I think this resource failed to get VersionID.

If you are concerned about error, I think regexp should be used..

It seems that VersionID will be checked only when using versioned_file.

[jyriok]

@cappyzawa "Sorry for the late reply."
no pb :)

so , i try your cmd :

$ aws s3api list-object-versions --bucket ee.cloud-backup --endpoint-url http://xxxxxxxxxx:8080

and :

{
    "DeleteMarkers": [
        {
            "Owner": {
                "DisplayName": "cloudfoundry-tls-ee",
                "ID": "cloudfoundry-tls-ee"
            },
            "IsLatest": true,
            "VersionId": ".ZRHO4SN-2KUk1gVk652QOO-Sg0z1.B",
            "Key": "backup/cf-backup-201804131453.tar",
            "LastModified": "2018-04-13T15:21:44.876Z"
        },
        {
            "Owner": {
                "DisplayName": "cloudfoundry-tls-ee",
                "ID": "cloudfoundry-tls-ee"
            },
            "IsLatest": true,
            "VersionId": "f5jvxkHt-AwDlxSSlMlfphBmTmCmyF.",
            "Key": "backup/director-backup-201804121326.tar",
            "LastModified": "2018-04-13T15:21:40.180Z"
        },
        {
            "Owner": {
                "DisplayName": "cloudfoundry-tls-ee",
                "ID": "cloudfoundry-tls-ee"
            },
            "IsLatest": true,
            "VersionId": "yQEjCYN52XVpERLsOznBsy4KXiQsUcZ",
            "Key": "backup/director-backup-201804122311.tar",
            "LastModified": "2018-04-13T15:21:42.598Z"
        }
    ],
    "Versions": [
        {
            "LastModified": "2018-04-13T14:53:27.642Z",
            "VersionId": "ki0oS7ISwPNpqGfUyjcezhXvrlFV-Oq",
            "ETag": "\"bebd98bc33036ffb57334f4e7957610e\"",
            "StorageClass": "STANDARD",
            "Key": "backup/cf-backup-201804131453.tar",
            "Owner": {
                "DisplayName": "cloudfoundry-tls-ee",
                "ID": "cloudfoundry-tls-ee"
            },
            "IsLatest": false,
            "Size": 1363456
        },
        {
            "LastModified": "2018-04-16T13:12:22.793Z",
            "VersionId": "WA8OqexsGknU5isTzbKXesWsF4gz3lI",
            "ETag": "\"81180c090c742c6ac4b6bc4ec7a0923d\"",
            "StorageClass": "STANDARD",
            "Key": "backup/cf-backup.tar",
            "Owner": {
                "DisplayName": "cloudfoundry-tls-ee",
                "ID": "cloudfoundry-tls-ee"
            },
            "IsLatest": true,
            "Size": 3646976
        },
        {
            "LastModified": "2018-04-16T12:22:24.730Z",
            "VersionId": "Kv9pqPx1ATf8xQGX997nt6BNRXsgxUu",
            "ETag": "\"e7cceafcbb2cf5f9922993d70d01bcfb\"",
            "StorageClass": "STANDARD",
            "Key": "backup/cf-backup.tar",
            "Owner": {
                "DisplayName": "cloudfoundry-tls-ee",
                "ID": "cloudfoundry-tls-ee"
            },
            "IsLatest": false,
            "Size": 3646976
        },
etc....

seem works :s
i use same account for s3 over concourse or aws cli

[cappyzawa]

@jyriok VersionID exists...
Well, version is stored after uploading.
VersionID exists in your screen shot.
Sorry, I mistake.

I can not confirm any more in my environment.

Does anyone know about this problem?

I think it strange that VersionID is judged to be empty...

[holmboe]

We are seeing this error as well and have not been able to find a solution. The S3-compatible service we are running against is Digital Ocean Spaces.

[jyriok]

if i go on the S3 resource task container, i can test to upload file to S3 and try to debug it. But do you have some sample s3 upload go script for done this or do i need to make my own with this libs based on this
https://medium.com/@questhenkart/s3-image-uploads-via-aws-sdk-with-golang-63422857c548
? (i don't know GO but i will learn :))

[marco-m]

Is this a duplicate of #23 ?

[xtreme-sameer-vohra]

@marco-m we believe that issue is more related to not having enough docs to specify one of regexp or versioned_file is required whereas this one is more about "s3-compatible" object stores having issues

One thing we noticed on another case involving the Ceph object store is that they weren't fully "s3-compatible"

1. The s3-resource delegates to the aws-sdk-client to do the heavy lifting for most of the logic
2. The s3 docs specify that the PUT Object operation must return a x-amz-version-id response header which will be used by the client to determine a versionId (https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPUT.html)
3. The Ceph docs did not mention anything about the response headers so we dug a bit deeper and found that particular header was not supported
   • https://github.com/ceph/ceph/blob/master/doc/dev/radosgw/s3_compliance.rst#common-response-headers
   • https://github.com/ceph/ceph/blob/master/doc/radosgw/s3.rst#unsupported-header-fields

So we've confirmed that without the x-amz-version-id header the resource was unable to resolve the version.

[xtreme-sameer-vohra]

One possible workaround is that instead of using the versioned_file configuration, using regexp instead will by pass checking that header.

s3-resource/out/command.go

Lines 81 to 89 in 8474109

	if request.Source.VersionedFile != "" {
	if versionID == "" {
	return Response{}, ErrObjectVersioningNotEnabled
	}
	version.VersionID = versionID
	} else {
	version.Path = remotePath
	}

[pnikonowicz]

we were able to fix this in S3 by enabling versioning on the bucket. it was disabled when we were getting this error.

to enable this, go into your bucket in S3 and click the Properties tab. There is a Versioning box needs to be listed as "enabled"