Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improved telemetry for worker initialization timeouts #10745

Open
satvu opened this issue Jan 15, 2025 · 0 comments
Open

Improved telemetry for worker initialization timeouts #10745

satvu opened this issue Jan 15, 2025 · 0 comments

Comments

@satvu
Copy link
Member

satvu commented Jan 15, 2025

It is difficult to pin down which methods or tasks are timing out during worker initialization. One example is that during a recent investigation, a timeout likely related to the wait between WorkerInitRequest and WorkerInitResponse bubbled up into a timeout error pointing at this line.

System.AggregateException : One or more errors occurred. (The operation has timed out.) ---> The operation has timed out.
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at async Microsoft.Azure.WebJobs.Script.WorkerFunctionMetadataProvider.GetFunctionMetadataAsync(IEnumerable`1 workerConfigs,Boolean forceRefresh) at /src/azure-functions-host/src/WebJobs.Script/Host/WorkerFunctionMetadataProvider.cs : 115 ---> (Inner Exception #0) System.TimeoutException : The operation has timed out.
    at Microsoft.Azure.WebJobs.Script.Grpc.GrpcWorkerChannel.PendingItem.OnTimeout() at /src/azure-functions-host/src/WebJobs.Script.Grpc/Channel/GrpcWorkerChannel.cs : 1799
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at async Microsoft.Azure.WebJobs.Script.Grpc.GrpcWorkerChannel.StartWorkerProcessAsync(CancellationToken cancellationToken) at /src/azure-functions-host/src/WebJobs.Script.Grpc/Channel/GrpcWorkerChannel.cs : 379<--- 

Potential next steps:

  • Review of worker initialization timeout flows
  • Review of the logging in this flow
  • Design proposal needed for improvements
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant