xread throws timeout errors when using block in asyncio mode

[kevinvalk]

Version:
redis-py: 5.0.1
redis: 7.0.12

Platform:
Docker containers (Linux) running on a MacBook M1

Description:
I believe this issues is literally #3008. I am using redis-py in asyncio mode. When I use xread with the block argument (block=1500) I get timeout exceptions that cancel my loop (I am listening to a redis stream).

Given that all works well with a small block=500 the behavior is quite weird. After some more investigation it seems to me that the current code that handles timeouts is not equipped to handle commands that block themselves. As in, no timeout parameter is provided so a real TimeoutError is raised.

redis-py/redis/asyncio/connection.py

Lines 582 to 589 in 6240ea1

	except asyncio.TimeoutError:
	if timeout is not None:
	# user requested timeout, return None. Operation can be retried
	return None
	# it was a self.socket_timeout error.
	if disconnect_on_error:
	await self.disconnect(nowait=True)
	raise TimeoutError(f"Timeout reading from {host_error}")

Given that #3008 sums it up pretty nicely maybe in the meantime some warnings in the docs and or code could help?

Timeout reading from 172.21.0.6:6379
Traceback (most recent call last):
  File "/venv/lib/python3.11/site-packages/redis/asyncio/connection.py", line 502, in read_response
    response = await self._parser.read_response(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/redis/_parsers/resp2.py", line 82, in read_response
    response = await self._read_response(disable_decoding=disable_decoding)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/redis/_parsers/resp2.py", line 90, in _read_response
    raw = await self._readline()
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/redis/_parsers/base.py", line 219, in _readline
    data = await self._stream.readline()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/streams.py", line 545, in readline
    line = await self.readuntil(sep)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/streams.py", line 637, in readuntil
    await self._wait_for_data('readuntil')
  File "/usr/local/lib/python3.11/asyncio/streams.py", line 522, in _wait_for_data
    await self._waiter
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/venv/lib/python3.11/site-packages/redis/asyncio/connection.py", line 501, in read_response
    async with async_timeout(read_timeout):
  File "/usr/local/lib/python3.11/asyncio/timeouts.py", line 111, in __aexit__
    raise TimeoutError from exc_val
TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/workspace/gateway/src/gateway/websockets/yjs.py", line 208, in _log_exception
    return await awaitable
           ^^^^^^^^^^^^^^^
  File "/workspace/gateway/src/gateway/websockets/yjs.py", line 145, in serve
    await redis.follower.xread(  # pyright: ignore[reportUnknownMemberType]
  File "/venv/lib/python3.11/site-packages/redis/asyncio/client.py", line 606, in execute_command
    return await conn.retry.call_with_retry(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/redis/asyncio/retry.py", line 62, in call_with_retry
    await fail(error)
  File "/venv/lib/python3.11/site-packages/redis/asyncio/client.py", line 593, in _disconnect_raise
    raise error
  File "/venv/lib/python3.11/site-packages/redis/asyncio/retry.py", line 59, in call_with_retry
    return await do()
           ^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/redis/asyncio/client.py", line 580, in _send_command_parse_response
    return await self.parse_response(conn, command_name, **options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/redis/asyncio/client.py", line 627, in parse_response
    response = await connection.read_response()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/redis/asyncio/sentinel.py", line 75, in read_response
    return await super().read_response(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/redis/asyncio/connection.py", line 520, in read_response
    raise TimeoutError(f"Timeout reading from {host_error}")
redis.exceptions.TimeoutError: Timeout reading from 172.21.0.6:6379

[kevinvalk]

You can work around this by eating the redis.exceptions.TimeoutError yourself (NOTE: this is not a normal TimeoutError so please make sure you use redis.exceptions.TimeoutError). However, you get around a maximum of 1s no matter how big you make the block parameter. OR if you use 0 to wait indefinitely.

So the real bug here is that the block parameter cannot reliable be used when providing a high number (above 250ms+-).