Fix intermittent Graph500 hang on ARM64 by forcing a stable libfabric provider#741
Merged
Merged
Conversation
… provider Graph500 runs as a single MPI rank and needs no network transport, yet on systems where MPICH selects the OFI (libfabric) 'sockets' provider (e.g. Ubuntu 22.04 / libfabric 1.11) that provider intermittently deadlocks in MPI_Ibarrier progress -- most often on ARM64 -- leaving the benchmark process spinning indefinitely with no output. Because the executor waits on the process with no per-process timeout, Virtual Client stalls until the outer step timeout, so the QoS PERF-GRAPH500 arm64 goal never completes. Root cause was confirmed on an ARM64 VM: a gdb backtrace of the hung process showed run_sssp -> aml_barrier -> MPI_Test spinning inside libfabric's sockets provider (epoll_pwait) while a single-rank MPI_Ibarrier never completes. Force the maintained 'tcp' provider (FI_PROVIDER=tcp) via the process beforeExecution hook when launching the workload binary. This bypasses the buggy sockets provider without changing results and applies to both linux-x64 and linux-arm64. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
ericavella
reviewed
Jul 10, 2026
ericavella
approved these changes
Jul 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix intermittent Graph500 hang on ARM64 (QoS_VC_ARM64_PERF-GRAPH500-linux-arm64)
Symptom
The QoS
PERF-GRAPH500arm64 goal never completes (0% pass). Virtual Clientlaunches the
graph500_reference_bfs_ssspbenchmark repeatedly for the profileduration; after tens of thousands of invocations one invocation hangs and never
returns. Because
Graph500Executorwaits on the process with no per-processtimeout, VC stalls silently until the outer Juno step timeout (~10h), and the
telemetry shows
Graph500Executor.ExecuteStopexactly one short ofExecuteStartwith no error emitted.Root cause (confirmed on ARM64 hardware)
Graph500 runs as a single MPI rank and needs no network transport, but on
Ubuntu 22.04 (MPICH 4.0 / libfabric 1.11) MPICH's OFI netmod falls through
to the deprecated
socketsprovider (verbs has no IPoIB; shmshm_openfails). That provider intermittently deadlocks in
MPI_Ibarrierprogress —most often on ARM64's relaxed memory model.
A
gdbbacktrace of a hung process (StateR, 100% CPU, no output) shows theexact stuck frame:
A single-rank
MPI_Ibarriernever completes, soMPI_Testreturnsflag=0forever while libfabric busy-spins. This is not a defect in the Graph500
source (the run is byte-for-byte deterministic: fixed seeds,
SCALE=10,EdgeFactor=16) — a minimal driver exercising only AML init/barrier/finalizeran 38,400× with no hang. The non-determinism lives in the MPI transport.
Fix
Force the maintained
tcplibfabric provider (FI_PROVIDER=tcp) on theworkload process via the existing
beforeExecutionhook. This bypasses thebuggy
socketsprovider, produces identical results, and applies to bothlinux-x64andlinux-arm64. No repackaging of the workload is required.Does this change what the benchmark measures? No.
Graph500 runs here as a single MPI rank (
num_mpi_processes = 1, anexplicitly captured metric in the workload docs),
so the MPI/AML collectives are effectively no-ops on a size-1 communicator and
FI_PROVIDERonly selects which libfabric provider services them. The measuredgraph-generation / BFS / SSSP kernels (TEPS) are untouched, which is why the
tcpruns produce correct, valid results. The workload docs describe Graph500purely as a data-intensive graph-traversal benchmark and never reference the
network transport as a measured dimension.
Validation
gdbbacktrace above (ARM64, MPICH 4.0 /libfabric 1.11) localizes the hang to the sockets-provider
MPI_Ibarrier.sockets;FI_PROVIDER=tcpselects a clean path with correct TEPS output.
FI_PROVIDER=tcpran 51,200 iterations at 64-way concurrency with0 hangs and correct results (past the production hang point of ~40,784).
Graph500ExecutorForcesAStableFabricProviderOnTheWorkloadProcess(x64 + arm64); all 9 Graph500 tests pass; project builds clean.
Follow-up (not in this PR)
Graph500Executorwaits on the benchmark with no per-process timeout, which iswhy a single wedged invocation stalls VC for the entire step. A per-process
timeout (as NTttcp/SockPerf/Latte/CPS executors already use) would convert any
future transport-level stall into a fast, diagnosable failure. Recommended as a
complementary hardening.