[BUG] Cancellation Tokens are being emitted for Queue trigger functions when entering Drain Mode

[nzthiago]

Library name and version

Microsoft.Azure.Functions.Worker.Extensions.Storage.Queues

Describe the bug

When function apps scale in on the Consumption, Flex consumption, and Elastic Premium plans, a drain mode request is sent to the host and extensions. Entering drain mode should not emit a cancellation token to executing functions, it should just stop the listeners so no new function executions happen on that instance. See this issue for a little more details: Azure/azure-webjobs-sdk#2795

It appears like the Microsoft.Azure.Functions.Worker.Extensions.Storage.Queues extension is emitting a cancellation token when drain mode is called, and this should not happen.

Requesting prioritizing fixing this as it has been addressed by other triggers. This is affecting customers on Flex Consumption as it is more aggressive to scale in, blocking some from going to production. Let me know if any more info is needed.

Expected behavior

Function app scale in gracefully and no cancellation token is emitted due to an instance being marked for scale in (drain mode).

Actual behavior

A cancellation token is being emitted when a function app instance gets marked for scale in (drain mode), causing function executions to be cancelled way earlier than they should.

Reproduction Steps

As a repro:

A simple function with a delay of 20 seconds, this is the only function in the app:

It has a 20 second delay in it. The app is deployed to a Flex Consumption azure functions hosting plan. It seems like drain mode is triggering the cancellation token (see in the logs that the token returns true), and this causes the message to appear in the queue again after a few minutes and this repeats again:

Environment

No response

[github-actions]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[nzthiago]

FYI this has already been addressed for other triggers, like Service Bus, Timer and others, hoping we could address it in a timely manner for queue storage as well.

[Peter-Juhasz]

I believe we see the same root cause here: Azure/azure-functions-dotnet-worker#2900

[nzthiago]

This PR from @amnguye was to address this and not emit a cancellation token for scale in (drain mode) for version 5.3.0 of the Microsoft.Azure.WebJobs.Extensions.Storage package and newer. Further investigation is needed as to why the cancellation token is still being emitted.

[FHajHusein]

Internal incident: IcM 579547346.

I believe the issue was introduced by PR 40792. The CancellationToken passed to ProcessMessageAsync is also specified for CompleteProcessingMessageAsync. When QueueListener.StopAsync is called, TaskSeriesTimer.Cancel is called, and the CancellationToken is canceled, thus canceling the deleting of the message by CompleteProcessingMessageAsync. The message eventually reappears. Before this PR, the CancellationToken wasn't set for CompleteProcessingMessageAsync.

It is also worth noting that currently, while stopping the listener, the execution CancellationToken is canceled if the drain mode is enabled, opposite to the expected behavior. I believe the condition should be changed to become as follows (but this of course wouldn't address the issue at hand):
if (!_drainModeManager?.IsDrainModeEnabled ?? true)

[amnguye]

Looking into the issue.

[seanmcc-msft]

This issue was fixed by #47791, and will be included in our next release in February.