Various S3 improvements #16
Conversation
* Add custom endpoint via `GOCACHE_AWS_URL`. * Add auth with session token via `GOCACHE_AWS_SESSION_TOKEN` * Use a default `us-east-1` region. * Fix ObjectID -> OutputID field rename * Fix TimeNanos -> Time field rename * Replace `GOCACHE_CACHE_KEY` with `GOCACHE_S3_PREFIX`. * Add compression to >8KB files on S3. * Use first 3 bytes of hash for prefix. * Fix NaN in timekeeping (and just ignore time not spent doing requests) * Update all dependencies
|
It is completely broken on Windows: golang/go#71059 RC1 also breaks if the experiment isn't enabled due to the renamed |
|
@or-shachar I was thinking, if there is a reason to (forcefully) separate on arch/platform. AFAICT the files are pretty much unique per platform anyway. We can either set it as metadata or tags (which will make it available to lifecycle filters). |
|
Thanks @klauspost ! Any chance you can pull changes from upstream, and add 1.24rc1 to the test matrix (github action files)? |
# Conflicts: # cachers/s3.go
|
Main merged and workflows updated, but I expect you need to approve them to run. |
|
mm... I was sure that I got the pull request trigger right. Can you kindly pull changes / rebase again? |
|
Merged + pushed. Seems like changes to workflows aren't being picked up before they are merged. |
|
Not sure why... I'll need to read a little more to figure it out. Will continue tomorrow. If you have an idea - let me know. |
| envVarS3BucketName = "GOCACHE_S3_BUCKET" | ||
| envVarS3CacheKey = "GOCACHE_CACHE_KEY" | ||
| envVarS3Prefix = "GOCACHE_S3_PREFIX" |
There was a problem hiding this comment.
Yes. See discussion above. Honestly I thought I was talking to you until I had already sent in the change.
| // Copyright 2009 The Go Authors. All rights reserved. | ||
| // Use of this source code is governed by a BSD-style | ||
| // license that can be found in the LICENSE file. |
There was a problem hiding this comment.
What is this ? external source file? why is it checked in here? should we import it from somewhere else?
There was a problem hiding this comment.
Forks
bytes.Bufferto make it seekable, since non-TLS requests will include checksum.
The aws SDK needs an input that is seekable to work with HTTP endpoints. So I forked bytes.Buffer to include that.
| if outputID == "" { | ||
| return "", errors.New("empty outputID") | ||
| } |
There was a problem hiding this comment.
Didn't test with EXPERIMENT option. For 1.24 it is.
This breaks the placement on older versions. It will probably also break when using versions prior to 1.24 RC.
GOCACHE_S3_URL.GOCACHE_S3_SESSION_TOKENus-east-1region.GOCACHE_CACHE_KEYwithGOCACHE_S3_PREFIX.bytes.Bufferto make it seekable, since non-TLS requests will include checksum.If you want to test on our playground you can try...