Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

snssqs: fix consumer starvation #3478

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Commits on Oct 15, 2024

  1. snssqs: fix consumer starvation

    The current implementation uses separate goroutines to process each
    message retrieved by `sqsClient.ReceiveMessageWithContext`, which is
    executed when `concurrencyMode` is set to `parallel`.
    The `consumeSubscription` function waits for all spawned goroutines to
    complete before receiving the next batch. However, this design has a
    critical flaw: it starves the consumer if new messages are published
    during the processing of an existing batch. These newly published
    messages won't be processed until all goroutines from the original batch
    have finished their work, effectively blocking the consumption of the
    new messages.
    
    This PR changes that by removing the waiting mechanism.
    
    Signed-off-by: Gustavo Chain <[email protected]>
    qustavo committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    3fc6313 View commit details
    Browse the repository at this point in the history
  2. snssqs: add concurrencyLimit param

    The `ConcurrencyLimit` param controls the number of concurrent process
    that handles messages.
    
    Signed-off-by: Gustavo Chain <[email protected]>
    qustavo committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    40078b7 View commit details
    Browse the repository at this point in the history
  3. Update pubsub/aws/snssqs/metadata.yaml

    Signed-off-by: Alessandro (Ale) Segala <[email protected]>
    ItalyPaleAle authored and qustavo committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    0c8b4d5 View commit details
    Browse the repository at this point in the history

Commits on Oct 16, 2024

  1. Configuration menu
    Copy the full SHA
    efcbd13 View commit details
    Browse the repository at this point in the history

Commits on Oct 25, 2024

  1. Configuration menu
    Copy the full SHA
    7827b51 View commit details
    Browse the repository at this point in the history