TimeOutException in HttpTrigger Function

[ansonpinhero]

I am encountering a TimeOutException at the following line of code:

azure-functions-dotnet-worker/extensions/Worker.Extensions.Http.AspNetCore/src/Coordinator/DefaultHttpCoordinator.cs

Line 36 in 2bbc2b9

return await contextRef.FunctionContextValueSource.Task.WaitAsync(TimeSpan.FromSeconds(FunctionContextTimeoutInSeconds));

This issue occurs intermittently. I have a Time Triggered function that invokes an HTTP Trigger function within the same Function App.

Below is the exception stack trace:

`Connection id "{ConnectionId}", Request id "{TraceIdentifier}": An unhandled exception was thrown by the application.

System.TimeoutException: The operation has timed out.
at Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore.WorkerRequestServicesMiddleware.Invoke(HttpContext context) in /mnt/vss/_work/1/s/extensions/Worker.Extensions.Http.AspNetCore/src/AspNetMiddleware/WorkerRequestServicesMiddleware.cs:line 28
at Datadog.Trace.ClrProfiler.AutoInstrumentation.AspNetCore.BlockingMiddleware.Invoke(HttpContext context) in /project/tracer/src/Datadog.Trace/ClrProfiler/AutoInstrumentation/AspNetCore/BlockingMiddleware.cs:line 115
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication1 application)

Could anyone provide insights or suggestions to debug and resolve this issue?

[fabiocav]

Hey @ansonpinhero, the exception above seems to indicate that either your worker didn't get the context information from the host, or there is something with the execution pipeline impacting the context synchronization.

How often are you seeing this happen? Can you tell whether this happens in specific scenarios like activation or scaling? If you can share your app name (or region, an invocation ID and a timeframe), we can also try to take a look a platform logs. If you're seeing this in production, opening a support request would also be a good way to proceed as your app information would be associated with that.

Thanks!

[ansonpinhero]

I have a Function App running in a Docker container on my local machine. I'm using Datadog for profiling and tracing. However, I am encountering this issue only when Datadog tracing is enabled (DD_TRACE_ENABLED = true).

According to the Datadog profiler's code, the exception is thrown when the WorkerRequestServicesMiddleware is invoked by a middleware injected by Datadog. Please see the following code snippet:

await _next(context).ConfigureAwait(false);

https://github.com/DataDog/dd-trace-dotnet/blob/47d26de6aeaf8b9552e9bfb148ac0061d1dd429f/tracer/src/Datadog.Trace/ClrProfiler/AutoInstrumentation/AspNetCore/BlockingMiddleware.cs#L115C17-L115C60

The test scenario involves sending HTTP requests, starting with 10 virtual users, ramping up to 100 virtual users, holding at that level for 10 minutes, and then ramping down to 0. The issue occurs when the load reaches 40 virtual users, where I start seeing exceptions.

Below is my function.

[Function("Hello")]
public IActionResult Run([HttpTrigger(AuthorizationLevel.Anonymous, "get", "post")] HttpRequest req)
{
    _logger.LogInformation("C# HTTP trigger function processed a request.");
    return new OkObjectResult("Welcome to Azure Functions!");
}

Dockerfile

FROM mcr.microsoft.com/azure-functions/dotnet-isolated:4-dotnet-isolated8.0 AS base
WORKDIR /home/site/wwwroot
EXPOSE 8080

FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build
ARG BUILD_CONFIGURATION=Release
WORKDIR /src
COPY ["src/TestFnApp/FunctionApp/.FunctionApp.csproj", "src/TestFnApp/FunctionApp/"]
RUN dotnet restore "./src/TestFnApp/FunctionApp/FunctionApp.csproj"
COPY . .
WORKDIR "/src/src/TestFnApp/FunctionApp"
RUN dotnet build "./FunctionApp.csproj" -c $BUILD_CONFIGURATION -o /app/build

FROM build AS publish
ARG BUILD_CONFIGURATION=Release
RUN dotnet publish "./FunctionApp.csproj" -c $BUILD_CONFIGURATION -o /app/publish /p:UseAppHost=false

FROM base AS final
WORKDIR /home/site/wwwroot
COPY --from=publish /app/publish .

COPY --from=datadog/serverless-init:1.2.6 /datadog-init /app/datadog-init
COPY --from=datadog/dd-lib-dotnet-init:v2.56 /datadog-init/monitoring-home/ /dd_tracer/dotnet/

ENV AzureWebJobsScriptRoot=/home/site/wwwroot \
    AzureFunctionsJobHost__Logging__Console__IsEnabled=true

CMD ["/app/datadog-init", "/azure-functions-host/Microsoft.Azure.WebJobs.Script.WebHost"]

[mike-at-enpal-de]

i was facing similar issue, in my case disabling aspnet health check middleware helped
(i completely removed services.AddHealthChecks() call)