[Enhancement] Adjust batch size in dispatcher so that a single batch doesn't exceed managedLedgerMaxReadsInFlightSizeInMB

[lhotari]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Current managedLedgerMaxReadsInFlightSizeInMB will have to be set to value that is higher than dispatcherMaxReadBatchSize * maxMessageSize.

Otherwise it could result in error Time-out elapsed while acquiring enough permits on the memory limiter to read from ledger [ledgerid], [topic], estimated read size [read size] bytes for [dispatcherMaxReadBatchSize] entries (check managedLedgerMaxReadsInFlightSizeInMB).
dispatcherMaxReadBatchSize defaults to 100 and maxMessageSize defaults to 5MB in bytes.

This means that currently managedLedgerMaxReadsInFlightSizeInMB must be set to 500 or higher to prevent the issue from occurring in all cases.

Solution

Adjust batch size in dispatcher so that a single batch doesn't exceed managedLedgerMaxReadsInFlightSizeInMB

Alternatives

No response

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[lhotari]

There's another problem. The current solution expects that managedLedgerReadEntryTimeoutSeconds is set. It will fail immediately without waiting in that case, resulting in a similar exception. I filed #23506 for this issue.