slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Repository	Step Name	Console log	Pull Request
3027321	dotnet-maui	RunOniOS_MauiDebug ARM64
1518340	dotnet/aspnetcore	Tests: macOS	Log
1518264	dotnet/xharness	OSX Debug	Log
1518162	dotnet/maui	Controls CollectionView	Log
1518143	dotnet/maui	Controls CollectionView	Log
3027103	dotnet-android	macOS > Tests > MSBuild+Emulator 11	Log
1518044	dotnet/aspnetcore	Tests: macOS	Log
1517948	dotnet/runtime	osx-arm64 release Installer_Build_And_Test	Log	dotnet/runtime#130825
1518022	dotnet/aspnetcore	Tests: macOS	Log
1503732	dotnet/maui	Controls Shell	Log
1517910	dotnet/maui	Controls Border,BoxView,Brush,Button	Log
3027037	dotnet-maui	Controls ViewBaseTests,VisualStateManager,Window	Log
1517876	dotnet/maui	Controls CollectionView	Log	dotnet/maui#36507
1517762	dotnet/maui	Controls CollectionView	Log	dotnet/maui#36672
3026966	dotnet-maui	Build macOS	Log
1517702	dotnet/maui	Controls CollectionView	Log
1517676	dotnet/maui	Controls Shell	Log	dotnet/maui#36631
1514711	dotnet/maui	Controls CollectionView	Log	dotnet/maui#36109
1517482	dotnet/maui	Controls CollectionView	Log	dotnet/maui#36657
1517234	dotnet/maui	Controls Layout	Log	dotnet/maui#36657
1517070	dotnet/maui	Controls CollectionView	Log
1516926	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
1517108	dotnet/xharness	Helix Tests Build_Debug	Log	dotnet/xharness#1640
1517089	dotnet/xharness	Helix Tests Build_Debug	Log	dotnet/xharness#1639
1516852	dotnet/maui	Controls CollectionView	Log	dotnet/maui#36654
1516832	dotnet/maui	Controls CollectionView	Log	dotnet/maui#36640
1516828	dotnet/maui	Controls CollectionView	Log	dotnet/maui#36657
1516733	dotnet/maui	Controls Shell	Log	dotnet/maui#36654
1516729	dotnet/maui	Controls CollectionView	Log	dotnet/maui#36657
1516484	dotnet/maui	Controls CollectionView	Log	dotnet/maui#36633
1516525	dotnet/maui	Controls CollectionView	Log	dotnet/maui#29952
3026083	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
1516428	dotnet/maui	Controls CollectionView	Log	dotnet/maui#36658
1516405	dotnet/maui	Controls CollectionView	Log	dotnet/maui#36413
1516409	dotnet/maui	Controls CollectionView	Log
3026018	dotnet-performance	Performance mlnet ubuntu 2204 arm64 Ampere 9.0	Log
1516322	dotnet/maui	Controls CollectionView	Log	dotnet/maui#36658
1516236	dotnet/maui	Controls CollectionView	Log	dotnet/maui#36413
1516210	dotnet/maui	Controls CollectionView	Log	dotnet/maui#36328
1516200	dotnet/maui	Controls CollectionView	Log	dotnet/maui#36170
3025840	dotnet-maui	Controls CollectionView	Log
1516192	dotnet/maui	Controls Shell	Log
3025733	dotnet-runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log
3025611	dotnet-sdk	TestBuild: macOS (x64)	Log	#62534
1516057	dotnet/maui	Controls CollectionView	Log
3025612	dotnet-sdk	TestBuild: macOS (x64)	Log	#62535
3025599	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
1515988	dotnet/runtime	maccatalyst-x64 Release AllSubsets_NativeAOT	Log
1515941	dotnet/maui	Controls Shell	Log
3025548	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
3025426	dotnet-sdk	TestBuild: macOS (x64)	Log	#62534
3025538	dotnet-sdk	Darwin_AoT_Tests	Log	#62664
3025425	dotnet-sdk	TestBuild: macOS (x64)	Log	#62535
3025313	dotnet-sdk	TestBuild: macOS (x64)	Log	#63003
3025369	dotnet-sdk	Darwin	Log	#62942
3025360	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
1515623	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
3025330	dotnet-maui	RunOniOS_MauiDebug x64
3025329	dotnet-android	macOS > Tests > MSBuild+Emulator 11	Log
3024986	dotnet-runtime	ios-arm64 Release AllSubsets_NativeAOT	Log	#62988
1515477	dotnet/maui	Controls CollectionView	Log	dotnet/maui#36633
3025314	dotnet-sdk	TestBuild: macOS (x64)	Log	#63004
1515360	dotnet/maui	Controls CollectionView	Log
1515232	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#130924
1515115	dotnet/aspnetcore	Tests: macOS	Log
1515054	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#130973
1515081	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
1515043	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#130825
1515014	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#130622
1514995	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#130881
1514750	dotnet/runtime	Libraries Test Run checked coreclr osx arm64 Debug	Log	dotnet/runtime#130966
1514887	dotnet/maui	Controls Shell	Log	dotnet/maui#35892
1514845	dotnet/maui	Controls Shell	Log	dotnet/maui#36631
1514847	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#130758
1511860	dotnet/maui	Controls CollectionView	Log	dotnet/maui#36472
1514729	dotnet/maui	Controls CollectionView	Log	dotnet/maui#36561
1514758	dotnet/maui	Controls Shell	Log	dotnet/maui#30875
1514505	dotnet/maui	Controls Image,ImageButton,IndicatorView,InputTransparent,IsEnabled,IsVisible	Log	dotnet/maui#36133
1514713	dotnet/maui	Controls Shell	Log	dotnet/maui#36577
3024988	dotnet-runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	#62989
1514744	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#130588
1514715	dotnet/maui	Controls CollectionView	Log	dotnet/maui#35846
1514640	dotnet/maui	Controls Shell	Log	dotnet/maui#36476
3024944	dotnet-performance	Performance sdk_scenarios windows Win11 x64 Viper 8.0	Log
1514659	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#130877
3024909	dotnet-android	macOS > Tests > MSBuild+Emulator 6	Log
1514618	dotnet/maui	Controls CollectionView	Log	dotnet/maui#36541
1514644	dotnet/aspnetcore	Tests: macOS	Log
1514579	dotnet/maui	Controls ScrollView	Log	dotnet/maui#35578
1514587	dotnet/runtime	osx-arm64 checked	Log	dotnet/runtime#130962
1514538	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#129319
1514501	dotnet/maui	Controls Layout	Log	dotnet/maui#36607
1514458	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
1514129	dotnet/runtime	maccatalyst-x64 Release AllSubsets_CoreCLR	Log
1514357	dotnet/aspnetcore	Tests: macOS	Log
1514303	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#130958
1514280	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#130935
1514259	dotnet/maui	Controls Layout	Log	dotnet/maui#36608
1514268	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#130957
1514255	dotnet/runtime	maccatalyst-x64 Release AllSubsets_CoreCLR	Log	dotnet/runtime#130691
Displaying 100 of 1323 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
24	342	1323

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.