Flaky simtest: test_repeated_node_crash blob-existence check fails deterministically per binary layout (0.75 != 1.0)

[halfprice]

Symptom

simtest_test_repeated_node_crash fails on CI with:

panicked at crates/walrus-simtest/src/test_utils.rs:555:
assertion left == right failed: left: 0.75, right: 1.0
(BlobInfoConsistencyCheck::check_blob_existence)

Seed 3, WALRUS_GRPC_MIGRATION_LEVEL=0. An Active node was recorded missing 25% of its certified blobs with no in-flight blob sync at some epoch.

Evidence that this is a pre-existing, layout-sensitive race (not a logic regression)

Observed on PR #3605, but bisection shows the failing and passing builds are behaviorally identical:

• PR head and its parent both fail identically (same seed, same test, same 0.75) — 4 CI runs total, including 3 reruns of the same job on different runners.
• The parent differs from an earlier same-day CI-green head only by identifier renames and comment rewrapping (audited token-by-token: no behavior change). Current main also passes the same job.
• Locally (macOS) the failing commit passes 10/10: seeds 1-8 of the test plus the full MSIM_TEST_SEED=3 WALRUS_GRPC_MIGRATION_LEVEL=0 cargo simtest simtest --profile simtest suite.

So the outcome is deterministic per (binary, platform) but flips when only symbol names/string literals change — a schedule lottery through binary layout, presumably via a pointer/layout-order-dependent iteration feeding the simulated schedule, interacting with msim's one non-hermetic channel (real RocksDB threads vs virtual time). This also matches the earlier flake-hunting attempt (ci: TEMP stress repeated_node_crash across 500 seeds, June 2026, branch shuo/adopt_new_package_management_test_with_fix).

Hypothesis for the underlying race

The existence check runs only when the node is Active and skips blobs in blob_sync_in_progress. A blob sync that fails and leaves retry scheduling (or is cancelled without marking) leaves its blob missing but absent from the in-progress set while the node is Active — exactly the observed signature. Worth auditing the blob-sync failure/retry path against the consistency check's snapshot ordering.

Repro pointers

• CI: https://github.com/MystenLabs/walrus/actions/runs/30886961344 (3 identical failures on 628d8e8), plus the parent bisect on 05fdac2.
• Local attempts (all passing) suggest reproducing requires the CI runner environment or the specific failing binary.

[github-actions]

Thank you for opening this issue, a team member will review it shortly. Until then, please do not interact with any users that claim to be from Sui or Walrus support and do not click on any links!

[terranweb3]

Thanks for the detailed investigation and reproduction notes.

I checked the report, but this does not appear to have a reliable external workaround. The fact that the same seed behaves differently depending on the CI binary layout strongly suggests a scheduler-sensitive race rather than a regression introduced by #3605.

The most suspicious area seems to be the blob-sync failure, cancellation, and retry lifecycle. In particular, a blob may become missing while the node is already Active, without remaining represented in blob_sync_in_progress.

I’m flagging this to the Walrus team for a closer audit. It may be useful to add tracing around:

• insertion and removal from blob_sync_in_progress
• sync task cancellation or failure
• retry scheduling
• transitions into Active
• the consistency-check snapshot

Thanks again for including the bisect results and exact CI run.

[halfprice]

Diagnostic trace captured (draft PR #3610, error-level logging, CI run 30904917477, seed 3):

• Failing node: node 7. At epoch 5 its Active-shard snapshot is [10, 11, 4] and blob ONiZyakfK8DzcnBEly-kVlUuTw1DprSLaQIfInVQPTY is missing at shard 4 (bloom pre-check and authoritative check both).
• Shard 4's history across snapshots: Active on node 5 at epoch 2 → Active on no node during epochs 3–4 (mid-sync on its then-owner) → Active on node 7 at epoch 5, missing the blob.
• No staking in this test: the shard movement is crash-induced committee change (10s epochs, 5–25s repeated crashes), so new owners sync from previous owners whose copies may themselves be incomplete (never reached Active).

So this is not a check-timing artifact: a shard genuinely reached Active while missing a certified blob, i.e., the shard-sync → missing-blob-detection → per-blob-recovery chain has a hole under the crash + shard-migration gauntlet.

Ruled out by code audit:

• Transfer batches are atomic (slivers + progress cursor + pending-recover entries in one RocksDB batch) — abort/crash cannot persist the cursor ahead of data.
• recover_blob writes the sliver before removing the pending entry — abort-safe in the safe direction.
• TODO(WAL-1050) (SST buffer tail loss) is not in play: sst_ingestion_config is None in tests.

Remaining candidates (need targeted diagnostics):

1. A blob certified while the destination shard is mid-transfer (ActiveSync): it is outside the transfer iterator's snapshot bound, so its data must arrive via the event-driven blob sync — if any path skips writing to a non-Active shard, the blob is lost when the shard flips Active.
2. recover_missing_blobs's not_certified skip removes a pending blob based on the local blob-info table — safe only if that table can never lag the shard-sync bound.
3. Source serving sync_shard_before_epoch from an incomplete (ActiveSync) copy: missing blobs are detected against the destination's local certified list, which should cover it — unless the epoch bounds disagree.

All failures are byte-identical across 5 CI runs (three heads × reruns) and unreproducible on macOS (10/10), consistent with the outcome being schedule-locked per (binary, platform).

[terranweb3]

Thanks for the additional details. I’ve shared this with the Walrus team for further investigation.

[halfprice]

Root cause identified (diagnostic rounds 2–5 on #3610)

The blob loss is real and the mechanism is now fully traced (seed 3, blob ONiZ…, shard 4, node 7):

1. Node 7 crashed and restarted with a stale replay position. Its last stream-processed epoch change was epoch 3 (19:29:15); at restart (~19:29:46) its local blob-info was replayed only to ~event index 526, while the on-chain epoch was already 5. Persisted node status was Active and the lag was under the catch-up threshold, so no recovery mode.
2. Startup started shard 4's sync with bound = on-chain epoch 5 (fresh cursor), while the node's certified-blob knowledge ended at ~epoch 3–4. Trace: sync phase entry shard=4 epoch_bound=5 resume_cursor=None at 19:29:46.8.
3. The transfer terminates when the destination's local certified list is exhausted (while next_blob_info.is_some() — and the early return when it is None). Blob ONiZ… (certified at epoch 4, event index 552 — not yet replayed) was invisible to the destination, sorts after its last known blob, and was therefore never requested, never fetched, and never pended. The sync completed in 49ms → shard Active.
4. Replay later reached the certified event (19:29:48.328) and the certified worker skipped the repair: current_event_epoch=Some(4) → is_stored_at_all_shards_at_epoch(blob, 4) → node 7's epoch-4 shards are {10, 11} → all stored → skip. The protocol's assumption — "the epoch-5 shard sync will transfer blobs for shards gained at 5" — was already violated by step 3.
5. Blob permanently missing from shard 4; the existence check (correctly — authoritative_says_stored=false, round 4) reports 0.75.

This is pre-existing on main: the old restart_syncs startup path used the same on-chain-epoch bound with the same destination-driven termination. The refactor branch merely re-rolled the deterministic schedule into the failing pocket on CI runners.

Proposed fix: the startup publication should derive its epoch (and the resulting sync bound) from the node's persisted latest-processed-event epoch, not the fetched on-chain epoch. Resumed syncs then match the node's actual knowledge; the replayed EpochChangeStart events re-quiesce and re-register at each boundary as designed, and the epoch-5 sync runs only after replay makes the node's certified list complete through epoch 5 — at which point the missing blob is enumerated and transferred or pended. (The recovery service already follows this discipline — it waits for the event epoch before running; shard sync does not.)

[halfprice]

Fix validated on CI. The event-epoch sync bound fix (e706bc8 in #3605, plus the startup membership gate in 7801cf6) passed the full simtest job — all five seeds including seed 3 with WALRUS_GRPC_MIGRATION_LEVEL=0, the configuration that reproduced the data loss identically in five consecutive pre-fix runs. Run: https://github.com/MystenLabs/walrus/actions/runs/30976534006

Summary of the landed fix:

• The shard-sync driver (transfers and metadata-recovery enumeration) bounds its work by current_event_epoch, awaiting it, instead of the fetched on-chain epoch — a restarted node's resumed syncs match its replay position, and replayed epoch changes advance the bound as the local certified-blob list becomes complete.
• The epoch-change executor records the event epoch inside the critical section before publishing, so reconciler-started syncs always observe the new epoch as their bound.
• A persisted-epoch startup seed was tried and withdrawn (8d55b3a, removed): replay re-processes events whose handling was not durably persisted, so a persisted event epoch can run ahead of the replayed state. The bounded startup delay (until the first replayed/new event sets the watch — at most well under an epoch given per-epoch system events) is the correct behavior.

The pre-existing hole this closes affects main's restart path as well; this issue can track porting the fix if #3605 is not merged soon.