Skip to content

Commit

Permalink
Fix: Chain writer failed due to TypeError
Browse files Browse the repository at this point in the history
Solution: Use `bytes` in memory everywhere. Not elegant but fixes the problem.

---

The Chain writer task for Chain.ETH failed with the following stacktrace:

```
aleph-pyaleph-1 | 2024-01-29 14:09:32 [ERROR] aleph.chains.connector: Chain writer task for Chain.ETH failed, relaunching in 10 seconds.
aleph-pyaleph-1 | Traceback (most recent call last):
aleph-pyaleph-1 |   File "/opt/pyaleph/src/aleph/chains/connector.py", line 63, in chain_writer_task
aleph-pyaleph-1 |     await connector.packer(config)
aleph-pyaleph-1 |   File "/opt/pyaleph/src/aleph/chains/ethereum.py", line 351, in packer
aleph-pyaleph-1 |     await self.chain_data_service.prepare_sync_event_payload(
aleph-pyaleph-1 |   File "/opt/pyaleph/src/aleph/chains/chain_data_service.py", line 75, in prepare_sync_event_payload
aleph-pyaleph-1 |     ipfs_cid = await self.storage_service.add_file(
aleph-pyaleph-1 |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
aleph-pyaleph-1 |   File "/opt/pyaleph/src/aleph/storage.py", line 293, in add_file
aleph-pyaleph-1 |     await self.add_file_content_to_local_storage(
aleph-pyaleph-1 |   File "/opt/pyaleph/src/aleph/storage.py", line 269, in add_file_content_to_local_storage
aleph-pyaleph-1 |     await self.storage_engine.write(filename=file_hash, content=file_content)
aleph-pyaleph-1 |   File "/opt/pyaleph/src/aleph/services/storage/fileystem_engine.py", line 26, in write
aleph-pyaleph-1 |     file_path.write_bytes(content)
aleph-pyaleph-1 |   File "/usr/lib/python3.11/pathlib.py", line 1066, in write_bytes
aleph-pyaleph-1 |     view = memoryview(data)
aleph-pyaleph-1 |            ^^^^^^^^^^^^^^^^
aleph-pyaleph-1 | TypeError: memoryview: a bytes-like object is required, not 'str'
```

It appears that a `StringIO` was passed to `aleph.storage.add_file(...)`, resulting in `file_content` being of type `str`, where the underlying function `aleph.services.storage.fileystem_engine.FileSystemStorageEngine.write` expects the type `bytes`.

Solution:
1. Encode the `archive_content` in UTF-8 before adding it in the storage service.
2. Modify the type annotations accordingly
3. Modify the signature of `aleph.storage.StorageService.add_file` to expect `bytes` instead of any `IO` type.
  • Loading branch information
hoh committed Jan 31, 2024
1 parent cad2d9c commit 19d2c51
Show file tree
Hide file tree
Showing 3 changed files with 6 additions and 9 deletions.
4 changes: 2 additions & 2 deletions src/aleph/chains/chain_data_service.py
Original file line number Diff line number Diff line change
Expand Up @@ -70,10 +70,10 @@ async def prepare_sync_event_payload(
messages=[OnChainMessage.from_orm(message) for message in messages]
),
)
archive_content = archive.json()
archive_content: bytes = archive.json().encode("utf-8")

ipfs_cid = await self.storage_service.add_file(
session=session, fileobject=StringIO(archive_content), engine=ItemType.ipfs
session=session, file_content=archive_content, engine=ItemType.ipfs
)
return OffChainSyncEventPayload(
protocol=ChainSyncProtocol.OFF_CHAIN_SYNC, version=1, content=ipfs_cid
Expand Down
4 changes: 2 additions & 2 deletions src/aleph/services/ipfs/service.py
Original file line number Diff line number Diff line change
Expand Up @@ -154,12 +154,12 @@ async def pin_add(self, cid: str, timeout: int = 30, tries: int = 1):
else:
break

async def add_file(self, fileobject: IO):
async def add_file(self, file_content: bytes):
url = f"{self.ipfs_client.api_url}add"

async with aiohttp.ClientSession() as session:
data = aiohttp.FormData()
data.add_field("path", fileobject)
data.add_field("path", file_content)

resp = await session.post(url, data=data)
return await resp.json()
Expand Down
7 changes: 2 additions & 5 deletions src/aleph/storage.py
Original file line number Diff line number Diff line change
Expand Up @@ -275,16 +275,13 @@ async def add_file_content_to_local_storage(
)

async def add_file(
self, session: DbSession, fileobject: IO, engine: ItemType = ItemType.ipfs
self, session: DbSession, file_content: bytes, engine: ItemType = ItemType.ipfs
) -> str:
if engine == ItemType.ipfs:
output = await self.ipfs_service.add_file(fileobject)
output = await self.ipfs_service.add_file(file_content)
file_hash = output["Hash"]
fileobject.seek(0)
file_content = fileobject.read()

elif engine == ItemType.storage:
file_content = fileobject.read()
file_hash = sha256(file_content).hexdigest()

else:
Expand Down

0 comments on commit 19d2c51

Please sign in to comment.