Intermittent Type-2 merge corruption in-process: ~1/1000 proposals produce an invalid block proof from valid inputs (reproduced under Shadow)

[GrapeBaBa]

Symptom

Roughly once per 600–1600 proposals, a proposer's buildBlockProof (Type-2 STARK merge via xmss_merge_type_1_to_type_2) returns success but produces a corrupt proof from verified-good inputs. Publish-time verification fails with error.InvalidBlockSignatures and the block is lost. Before #986, one such loss froze an entire 32-node simulation permanently (the slot-463 incident: the lost block had advanced the proposer's checkpoint, putting all other 31 nodes into peers_materially_ahead, which silenced every duty with no escape).

Observed so far only under Shadow simulation (32-node, 4s slots, prover cost modeled in virtual time); real-network runs have not yet accumulated enough proposals to expect the event.

Evidence chain (instrumented runs + offline re-verification)

Instrumentation: on publish failure the node dumps the full merge inputs (Type-1 wires, participant pubkeys, proposer sig) plus the produced SignedBlock to disk; an offline tool re-verifies/re-merges from the dump.

For every captured failure (4 samples: slots 621, 249, 342, 1110 across two runs):

1. the dumped Type-2 fails verification offline → the bytes are genuinely bad, the live verify was correct to reject;
2. every Type-1 input verifies standalone → inputs good;
3. re-merging the exact same inputs offline → valid proof, 100% of attempts → the live in-process merge corrupted its output;
4. the prover is fully deterministic (two offline re-merges are byte-identical), so corrupt-vs-good diffs give an exact corruption map:
   • lz4 wire intact, postcard head (infos) intact;
   • first divergence at a consistent structural position (~11% into the plaintext, ~26–28 KB; one sample at ~0.3%), everything after cascades (Fiat-Shamir);
   • the corrupt region is well-formed varint data with different values — i.e. the prover honestly continued from one wrong internal read;
5. heap dependence: pinning large allocations onto the recycled heap (mallopt(M_MMAP_THRESHOLD)) moved the first failure from slot ~460+ to slot 249 — the natural ~400-slot onset matches glibc's dynamic mmap-threshold warmup, after which prover buffers start landing on recycled blocks;
6. heap poisoning (M_PERTURB) did not amplify the rate → not an always-taken-path uninitialized read;
7. concurrency probes at failure time: no concurrent STARK verifies in flight, but the 1–2s merge window overlaps the per-slot burst of raw XMSS sign/verify on other threads sharing the process heap.

Ruled out (source audit at leanVM e2592df4 + experiments)

• zk-alloc arena: dead code on this path (enable_arena never called by multisig-glue; begin_phase only in benchmarks);
• thread-pool sharing: leanVM uses its own fixed pool, leansig raw ops use global rayon — disjoint thread sets;
• prove-path globals: twiddles RwLock-protected, bytecode OnceLock, PoW result re-validated;
• wire/copy-layer corruption, input corruption, zeam-side buffer handling (offline tool exercises the same FFI surface).

Open suspects (the corrupting writer is not yet identified)

1. a rare-path memory bug inside the leanVM prover (uninit/OOB beyond the audited hot paths);
2. an out-of-bounds writer elsewhere in the shared process (zeam/glue/leansig) landing in prover-owned heap;
3. a Shadow artifact — the simulator's shim lives in the same address space, and all reproductions are under Shadow.

Discriminating next step: a native (non-Shadow) soak of ~2000+ slots with the same dump instrumentation. Native reproduction → upstream leanVM issue with the repro kit + consider process isolation for the merge (leanVM's own docs recommend process-level parallelism); no native reproduction → Shadow-compat note only.

Mitigations in flight

• fix(node): material-lead sync gating to prevent duty-freeze on small finalization leads #986 — a 1-slot peer finalization lead no longer silences duties: one lost block now self-heals in ~1 slot instead of freezing the network (verified by a second incident with identical loss but no checkpoint lead, which self-healed).
• fix(node): propose-path layering + publish latency + coverage report #988 — removes the redundant publish-hop re-verify/re-transition (unrelated to this corruption, but the verify that catches it sits there today; peers' gossip-import verification still rejects corrupt proofs regardless).

Repro kit (instrument branch with dump-on-fail hooks, the offline classifier/differ tools, and 4 proof dumps) is available — ping me.