right_awsbase.rb - a '?' character in S3 URL results in bad URL

[tomandersen]

If you use aws to generate a URL with an expiry for S3:

s3 = Aws::S3.new(aws_id, aws_secret)
bucket_gen = Aws::S3Generator::Bucket.create(s3, bucket)
# create a signed url good for a while

url = bucket_gen.get("key with ? in it", 30.days)
the resulting URL will not work: There are two problems:

1. The ? does not get encoded to %3F (it should)
2. The resulting bad URL also starts the argument list with '&' instead of ?, presumably because some bit of software see that there is already a ? and just adds the args as extras.

Speculation and guesswork:
I looked at right_awsbase.rb it seems that the problem lies in the amz_escape

eg: add this around line 160
e2 = e2.gsub("?", "%3F")

Or maybe you need to use different encoding for the key from the other arguments.

I have much the same program accessing the same files written in Objective-C. There I (well ASI) encode the S3Key only with lots of encoding. It really can't hurt, except for readability of the resulting URL (so I don't encode for example /)

CFURLCreateStringByAddingPercentEscapes(kCFAllocatorDefault, (CFStringRef)key, NULL, CFSTR(":?#[]@!$ &'()*+,;="<>%{}|^~`"), NSUTF8StringEncoding);

I tried encoding the ? in the key before passing it in, but that did not work : the %3F then got double coded.

An S3 Key can be any string of utf-8 bytes.

You should likely also test with a string with lots of weird chars, just to be sure...

Thanks, the tools seem to work quite well.

--Tom Andersen

[tomandersen]

Oh forgot to mention that I tested this in right_aws 2.0.0 and had the same issue. At least I'm pretty sure I did. I have only been using ruby for three weeks --Tom

[tomandersen]

Correction - it is right-awsbase.rb::URLEncode() that is the problem:
Adding the line below fixes the problem for the ? symbol.
S3 Keys can be any utf8 sequence.
e.gsub(/?/, "%3F")
I think that it is tricky to figure out the other characters that should be encoded for the s3Key, and is this the same set as for SQS? It seems that URLEncode is only used to encode S3 keys and Something called the SQS service.

http://developer.amazonwebservices.com/connect/thread.jspa?threadID=47863&tstart=0

[tomandersen]

So I went looking through the source code for libs3 version 1.4. posted on Amazon.

In that code, a function called urlEncode in util.c encodes urls as follows:

All characters except a-Z, 0-9 and urlSafe

• _    .     !    ~   \*   '    (    )    /  
  

are encoded. I would guess that this is likely a good way to do it.

[tomandersen]

url safe is actually (i think a bullet was added instead of a minus sign)

" - _ . ! ~ * ' ( ) / "

--Tom

[tomandersen]

If fixed the bug in a fork. Now generates urls, etc that work with ?, + and all sorts of weird characters in the s3 key.

Note that I did not change how SQS encodes URLs, as I can't test that.

http://github.com/tomandersen/aws