Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python3 needs mode rb in upload_file #38

Open
neilmb opened this issue Apr 6, 2018 · 0 comments · May be fixed by #39
Open

Python3 needs mode rb in upload_file #38

neilmb opened this issue Apr 6, 2018 · 0 comments · May be fixed by #39

Comments

@neilmb
Copy link
Contributor

neilmb commented Apr 6, 2018

I got this error trying to upload_file under Python 3:

Python 3.5.2 (default, Nov 23 2017, 16:37:01)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import s3po
>>> conn = s3po.Connection.s3(aws_access_key_id='************', aws_secret_access_key='*********')
>>> conn.upload_file('*********', 'sbox/test', 'setup.py')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/vagrant/s3po/connection.py", line 72, in upload_file
    headers=headers, extra=extra, retries=retries)
  File "/vagrant/s3po/connection.py", line 63, in upload
    bucket, key, obj_or_data, retries=retries, **opts)
  File "/vagrant/s3po/backends/s3.py", line 54, in upload
    key.upload_fileobj(source, Config=config, ExtraArgs=extra)
  File "/home/ubuntu/.local/lib/python3.5/site-packages/boto3/s3/inject.py", line 513, in object_upload_fileobj
    ExtraArgs=ExtraArgs, Callback=Callback, Config=Config)
  File "/home/ubuntu/.local/lib/python3.5/site-packages/boto3/s3/inject.py", line 431, in upload_fileobj
    return future.result()
  File "/home/ubuntu/.local/lib/python3.5/site-packages/s3transfer/futures.py", line 73, in result
    return self._coordinator.result()
  File "/home/ubuntu/.local/lib/python3.5/site-packages/s3transfer/futures.py", line 233, in result
    raise self._exception
  File "/home/ubuntu/.local/lib/python3.5/site-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/home/ubuntu/.local/lib/python3.5/site-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/home/ubuntu/.local/lib/python3.5/site-packages/s3transfer/upload.py", line 692, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "/home/ubuntu/.local/lib/python3.5/site-packages/botocore/client.py", line 324, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/ubuntu/.local/lib/python3.5/site-packages/botocore/client.py", line 603, in _make_api_call
    request_signer=self._request_signer, context=request_context)
  File "/home/ubuntu/.local/lib/python3.5/site-packages/botocore/hooks.py", line 242, in emit_until_response
    responses = self._emit(event_name, kwargs, stop_on_response=True)
  File "/home/ubuntu/.local/lib/python3.5/site-packages/botocore/hooks.py", line 210, in _emit
    response = handler(**kwargs)
  File "/home/ubuntu/.local/lib/python3.5/site-packages/botocore/handlers.py", line 210, in conditionally_calculate_md5
    calculate_md5(params, **kwargs)
  File "/home/ubuntu/.local/lib/python3.5/site-packages/botocore/handlers.py", line 188, in calculate_md5
    binary_md5 = _calculate_md5_from_file(body)
  File "/home/ubuntu/.local/lib/python3.5/site-packages/botocore/handlers.py", line 202, in _calculate_md5_from_file
    md5.update(chunk)
TypeError: Unicode-objects must be encoded before hashing
>>>

The problem is that https://github.com/seomoz/s3po/blob/master/s3po/connection.py#L69 opens the file in unicode mode so that the file object that trickles down to boto is giving out Unicode objects instead of bytes.

The obvious fix is to use rb when we open that file, since we want to upload a byte-exact copy of the file.

@neilmb neilmb linked a pull request Apr 6, 2018 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant