slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
930754	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46316
931251	maestro-auth-test/helix-queue-insights-test	HelixTests	Log
931167	dotnet/sdk	OSX_x64	Log	dotnet/sdk#46342
931151	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46183
931139	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46343
930919	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46244
931118	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46342
931105	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46281
930992	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#111684
931065	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#111311
930972	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#111775
930980	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
930960	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#111595
931035	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#111722
931024	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#110053
931026	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#111023
931006	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#111871
930997	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#111872
929953	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#111024
930888	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#111867
930956	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#109673
930946	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#98643
930951	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46261
930936	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#110641
930823	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#111636
930790	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#110827
930899	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46340
930866	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#111798
930862	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#111215
930860	dotnet/sdk	OSX_arm64	Log	dotnet/sdk#46063
930859	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46063
930350	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#109164
930809	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#111751
930782	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46338
930769	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#111869
928878	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46289
930673	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#107115
930661	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#111584
930734	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#45836
930730	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries		dotnet/runtime#111710
930718	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46329
930642	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#111691
930640	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46333
930637	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46330
930628	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#111852
930633	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46334
930619	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#108732
930606	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#111864
930568	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#111858
930499	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46244
930490	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#111434
930423	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#111706
930435	dotnet/xharness	Helix Tests Build_Debug	Log
930413	dotnet/runtime	osx-arm64 Release CoreCLR_Release	Log
930412	dotnet/runtime	osx-arm64 Release CoreCLR_Release	Log
930403	dotnet/runtime	osx-arm64 Release CoreCLR_Release	Log
930379	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#111859
930374	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#110566
930317	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#111853
930338	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#111858
930014	dotnet/xharness	Helix Tests Build_Debug	Log	dotnet/xharness#1352
930100	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#111327
930303	dotnet/xharness	Helix Tests Build_Debug	Log	dotnet/xharness#1345
930278	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
930300	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#111856
930252	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#111852
930196	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#111849
930233	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#111743
930222	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#111850
929984	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#42416
930177	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#111328
930172	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#111826
930170	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#111692
930162	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#111518
930160	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#111690
930165	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#111691
930144	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#111606
2628933	dotnet-binaryen	macOS_x64	Log
930081	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#111848
2628932	dotnet-cpython	macOS_arm64
930084	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#111604
930158	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46319
930152	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46318
930151	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46195
930133	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46316
930120	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46315
930103	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46314
929992	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46258
929973	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#111846
929407	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46290
929680	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46259
929679	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46195
929410	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46281
929217	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46261
929172	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46300
929672	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#46306
929942	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#110827
929957	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#111598
929906	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#111843
929899	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
Displaying 100 of 1135 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
89	280	1135

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.