Skip to content

Fix intermittent Graph500 hang on ARM64 by forcing a stable libfabric provider#741

Merged
ericavella merged 1 commit into
mainfrom
users/alexwill/graph500-arm64-fi-provider
Jul 10, 2026
Merged

Fix intermittent Graph500 hang on ARM64 by forcing a stable libfabric provider#741
ericavella merged 1 commit into
mainfrom
users/alexwill/graph500-arm64-fi-provider

Conversation

@AlexWFMS

@AlexWFMS AlexWFMS commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Fix intermittent Graph500 hang on ARM64 (QoS_VC_ARM64_PERF-GRAPH500-linux-arm64)

Symptom

The QoS PERF-GRAPH500 arm64 goal never completes (0% pass). Virtual Client
launches the graph500_reference_bfs_sssp benchmark repeatedly for the profile
duration; after tens of thousands of invocations one invocation hangs and never
returns. Because Graph500Executor waits on the process with no per-process
timeout, VC stalls silently until the outer Juno step timeout (~10h), and the
telemetry shows Graph500Executor.ExecuteStop exactly one short of
ExecuteStart with no error emitted.

Root cause (confirmed on ARM64 hardware)

Graph500 runs as a single MPI rank and needs no network transport, but on
Ubuntu 22.04 (MPICH 4.0 / libfabric 1.11) MPICH's OFI netmod falls through
to the deprecated sockets provider (verbs has no IPoIB; shm shm_open
fails). That provider intermittently deadlocks in MPI_Ibarrier progress —
most often on ARM64's relaxed memory model.

A gdb backtrace of a hung process (State R, 100% CPU, no output) shows the
exact stuck frame:

#0  epoll_pwait (timeout=0)
#1-4 libfabric.so.1        (OFI 'sockets' provider)
#5-6 libmpich.so.12
#7  PMPI_Test
#8  aml_barrier            (while(flag==0){ MPI_Test(&hndl,&flag); aml_poll(); })
#9  run_sssp
#10 main

A single-rank MPI_Ibarrier never completes, so MPI_Test returns flag=0
forever while libfabric busy-spins. This is not a defect in the Graph500
source (the run is byte-for-byte deterministic: fixed seeds, SCALE=10,
EdgeFactor=16) — a minimal driver exercising only AML init/barrier/finalize
ran 38,400× with no hang. The non-determinism lives in the MPI transport.

Fix

Force the maintained tcp libfabric provider (FI_PROVIDER=tcp) on the
workload process via the existing beforeExecution hook. This bypasses the
buggy sockets provider, produces identical results, and applies to both
linux-x64 and linux-arm64. No repackaging of the workload is required.

Does this change what the benchmark measures? No.

Graph500 runs here as a single MPI rank (num_mpi_processes = 1, an
explicitly captured metric in the workload docs),
so the MPI/AML collectives are effectively no-ops on a size-1 communicator and
FI_PROVIDER only selects which libfabric provider services them. The measured
graph-generation / BFS / SSSP kernels (TEPS) are untouched, which is why the
tcp runs produce correct, valid results. The workload docs describe Graph500
purely as a data-intensive graph-traversal benchmark and never reference the
network transport as a measured dimension.

Validation

  • Root cause: authoritative gdb backtrace above (ARM64, MPICH 4.0 /
    libfabric 1.11) localizes the hang to the sockets-provider MPI_Ibarrier.
  • Provider check: default env falls through to sockets; FI_PROVIDER=tcp
    selects a clean path with correct TEPS output.
  • Stress: FI_PROVIDER=tcp ran 51,200 iterations at 64-way concurrency with
    0 hangs and correct results (past the production hang point of ~40,784).
  • Unit tests: added Graph500ExecutorForcesAStableFabricProviderOnTheWorkloadProcess
    (x64 + arm64); all 9 Graph500 tests pass; project builds clean.

Follow-up (not in this PR)

Graph500Executor waits on the benchmark with no per-process timeout, which is
why a single wedged invocation stalls VC for the entire step. A per-process
timeout (as NTttcp/SockPerf/Latte/CPS executors already use) would convert any
future transport-level stall into a fast, diagnosable failure. Recommended as a
complementary hardening.

… provider

Graph500 runs as a single MPI rank and needs no network transport, yet on
systems where MPICH selects the OFI (libfabric) 'sockets' provider (e.g.
Ubuntu 22.04 / libfabric 1.11) that provider intermittently deadlocks in
MPI_Ibarrier progress -- most often on ARM64 -- leaving the benchmark process
spinning indefinitely with no output. Because the executor waits on the
process with no per-process timeout, Virtual Client stalls until the outer
step timeout, so the QoS PERF-GRAPH500 arm64 goal never completes.

Root cause was confirmed on an ARM64 VM: a gdb backtrace of the hung process
showed run_sssp -> aml_barrier -> MPI_Test spinning inside libfabric's sockets
provider (epoll_pwait) while a single-rank MPI_Ibarrier never completes.

Force the maintained 'tcp' provider (FI_PROVIDER=tcp) via the process
beforeExecution hook when launching the workload binary. This bypasses the
buggy sockets provider without changing results and applies to both linux-x64
and linux-arm64.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@ericavella ericavella enabled auto-merge (squash) July 10, 2026 18:02
@ericavella ericavella merged commit 08a89d8 into main Jul 10, 2026
5 checks passed
@ericavella ericavella deleted the users/alexwill/graph500-arm64-fi-provider branch July 10, 2026 18:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants