Function calls get cancelled after a few seconds-minutes even though timeout is explicitly set

[Peter-Juhasz]

Our Function App is unavailable due to continuously being cancelled by the host after a few seconds or minutes, while our work items take minutes, sometimes even half an hour to process.

Plan: Flex Consumption Plan
Region: North Europe
Application Runtime: .NET 9
Functions Runtime: V4, dotnet-isolated .NET 9

Default timeout: 30 minutes
Timeout explicitly set by our app: 30 minutes

Host.json file:

{
	"version": "2.0",
	"functionTimeout": "00:30:00",
	"extensions": {
		"queues": {
			"maxPollingInterval": "00:00:01",
			"messageEncoding": "none",
			"maxDequeueCount": 1,
			"batchSize": 1
		}
	}
}

Reproduction:

	[Function("ProcessTest")]
	public async Task RunDocumentPipelineAsync(
		[QueueTrigger("work-items", Connection = "Queues")]
		WorkItem item,
		FunctionContext context,
		CancellationToken cancellationToken
	)
	{
		logger.LogInformation("Started {Id} at {time}", context.InvocationId, timeProvider.GetUtcNow());
		cancellationToken.Register(() =>
		{
			logger.LogInformation("Cancelled {Id} at {time}", context.InvocationId, timeProvider.GetUtcNow());
		});

		// work and pass cancellationToken ...
		await DoWorkAsync(item, cancellationToken);
	}

Logs:

Started edead22a-3ac6-489b-b092-9904e267abe4 at 12/14/2024 14:30:08 +00:00
Cancelled edead22a-3ac6-489b-b092-9904e267abe4 at 12/14/2024 14:31:19 +00:00

We have also reported this issue to Azure Support, because our app is unavailable due to this platform issue.
Azure Support ticket ID: 2412140050000384

Also reported by others:

Hi!
I have a similar issue using Flex Consumption Plan with Queue Trigger. In the middle of the exectuion after 30-40 secs the function host requests a cancellation so my function shuts down. To solve this I had to pass CancellationToken.None to the downstream calls instead of the real CancellationToken coming from the function entry.

Originally posted by @bkulcsar in #2510

Workaround?

While @bkulcsar mentioned passing CancellationToken.None to downstream calls as a workaround, it may make the situation even worse. Because if the application does not shutdown gracefully after it got the signal, the host may just kill the process without any further warning, interrupting not only processing, but losing all telemetry with it, and losing sight of failures.