Error when trying to upload to Synapse using a manifest

[BryanFauble]

For the specified versions of Python/associated packages (synapseclient 2.3.1,Python==3.8,3.9, pandas>=0.25.0,<1.5, boto3>=1.7.0,<2.0), I seemed to run into the following error when trying to upload to Synapse using a manifest I created:

Starting upload...
cached_time: {'modified_time': '2024-10-01T22:21:16.000Z', 'content_md5': '26ba87912a307ec8a5b5b2ee4231bc18'}
cached_time: {'modified_time': '2024-10-01T20:32:31.000Z', 'content_md5': 'a752cdabbc916a533309ff418112f77e'}
cached_time: {'modified_time': '2024-10-01T22:20:49.000Z', 'content_md5': '46e9b576d19ad1f72dbb884eea39eb35'}
cached_time: {'modified_time': '2024-10-01T21:31:39.000Z', 'content_md5': 'e511738ed4ae16dbf1cbbb3328dd3b76'}
cached_time: {'modified_time': '2024-10-01T22:21:23.000Z', 'content_md5': '6986f185d6059db4ace0074606fa83fd'}
cached_time: {'modified_time': '2024-10-01T21:24:36.000Z', 'content_md5': '935ccb2f6ae7c9a8bef672a9f913f0b5'}
Traceback (most recent call last):
  File "/usr/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/xloinaz/GDC_HCMI_analyses/mutational_signature_analysis/sga_venv_python_3.9/lib/python3.9/site-packages/synapseutils/sync.py", line 570, in _upload_item
    entity = self._syn.store(item.entity, used=used, executed=executed, **item.store_kwargs)
  File "/home/xloinaz/GDC_HCMI_analyses/mutational_signature_analysis/sga_venv_python_3.9/lib/python3.9/site-packages/synapseclient/client.py", line 1048, in store
    or not self.cache.contains(bundle['entity']['dataFileHandleId'], entity['path'])
  File "/home/xloinaz/GDC_HCMI_analyses/mutational_signature_analysis/sga_venv_python_3.9/lib/python3.9/site-packages/synapseclient/core/cache.py", line 146, in contains
    return compare_timestamps(_get_modified_time(path), cached_time)
  File "/home/xloinaz/GDC_HCMI_analyses/mutational_signature_analysis/sga_venv_python_3.9/lib/python3.9/site-packages/synapseclient/core/cache.py", line 59, in compare_timestamps
    if cached_time.endswith(".000Z"):
AttributeError: 'dict' object has no attribute 'endswith'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/xloinaz/GDC_HCMI_analyses/mutational_signature_analysis/sga_venv_python_3.9/lib/python3.9/site-packages/synapseutils/monitor.py", line 49, in with_retry_and_messaging
    output = func(*args, **kwargs)
  File "/home/xloinaz/GDC_HCMI_analyses/mutational_signature_analysis/sga_venv_python_3.9/lib/python3.9/site-packages/synapseutils/sync.py", line 919, in _manifest_upload
    uploader.upload(items)
  File "/home/xloinaz/GDC_HCMI_analyses/mutational_signature_analysis/sga_venv_python_3.9/lib/python3.9/site-packages/synapseutils/sync.py", line 496, in upload
    self._abort(futures)
  File "/home/xloinaz/GDC_HCMI_analyses/mutational_signature_analysis/sga_venv_python_3.9/lib/python3.9/site-packages/synapseutils/sync.py", line 471, in _abort
    raise ValueError("Sync aborted due to upload failure") from exception
ValueError: Sync aborted due to upload failure

I manually fixed this bug in the source code by instead setting "cached_time = cached_time['modified_time']", but even after I do that I run into the issues "botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied" and "The project storage usage exceeds the limit for the storage location (Project: syn26004575, Storage Location: 1, Usage: 125.61 GiB, Limit: 100 GiB)." Is there anything that can be done about these three issues? For the second issue, is this just a permissions issue I need to sort out with the AWG? For the third issue, is it no possible to store more than 100GB in a given folder with nested contents? Thanks!

Originally posted by @xaviloinaz in #902

[BryanFauble]

@xaviloinaz

AttributeError: 'dict' object has no attribute 'endswith'

This error occurs when you run a newer and older version of the Synapse Python Client. There are changes in v3.2.0 of the client that modify the on-disk cache to be in a different format causing the AttributeError: 'dict' object has no attribute 'endswith' error. Since you've run a newer version of the client you should continue to do so, or as you found out, you'll run into this error and need to provide the hack you had done cached_time = cached_time['modified_time']

I run into the issues "botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied"

I suspect that this is a permissions issue with the bucket that you're using. Could you double check your permissions, and if you're still running into issues, provide some more context and/or scripts around this?

"The project storage usage exceeds the limit for the storage location (Project: syn26004575, Storage Location: 1, Usage: 125.61 GiB, Limit: 100 GiB)."

Synapse now limits the free plan to <100 GiB of content as detailed on this page: https://www.synapse.org/Plans:default .