Speed up chain compaction, harden PIBD validation, crash-safe compact by iho · Pull Request #3906 · mimblewimble/grin

iho · 2026-07-10T18:10:10Z

Three related fixes for low-end hardware reliability and post-PIBD stability, plus a stress harness for kill-during-compact.

Compaction speed (#3872)

Avoid quadratic scan when rewriting PMMR files. write_tmp_pruned checked prune_pos.contains for every element against the whole remaining prune list. Both are sorted, so this was O(elements × prune_count). Now only the head of the remaining slice is compared: O(elements + prune_count).

Bound the header walk in init_output_pos_index. When any output was missing from the index, the rebuild walked every height from genesis to head and kept walking after all missing outputs were placed. It now binary-searches for the first relevant header and breaks once the last missing entry is indexed.

PIBD segment hash validation (PR #3820 fallout)

Segment root validation only checked hashes used to rebuild the segment root, while apply writes every carried hash into the local PMMR. A peer could pass a valid root proof with poisoned intermediate hashes; failure only appeared at full state validation as a bad root (PIBD restart loop).

After the merkle proof succeeds, every carried hash must be bound to that root (consumed during reconstruction, or a child of a bound parent that hashes correctly). Dense segments from uncompacted peers still pass when consistent; unbound/wrong hashes fail early as UnverifiableHash.

Crash-safe compaction

A hard kill between replacing hash/data and flushing the prune list left compacted files with a stale prune list → wrong shifts → bad root on restart. Compaction now stages all new files, fsyncs, writes a journal marker, then renames. On open, a present journal rolls renames forward; temps without a journal are discarded as abandoned prep.

QEMU / weak-VPS stress harness

etc/qemu-compact-stress/ + store/examples/compact_crash_stress.rs:

Build a large pruned PMMR and time compaction (slow path under load)
SIGKILL the compact process at hash_tmp / data_tmp / journal and reopen
Modes: native, qemu-user (TCG), Docker linux/amd64 with 0.5 CPU / 512MB (QEMU on arm hosts)
Env-gated pause hooks in check_compact (no-op unless set)

./etc/qemu-compact-stress/run.sh              # native kill at journal
./etc/qemu-compact-stress/run.sh --all-phases
./etc/qemu-compact-stress/run.sh --weak       # weak VPS via Docker+QEMU

Manual workflow: .github/workflows/compact-stress.yaml (workflow_dispatch).

Testing

cargo test -p grin_store -p grin_core -p grin_chain
segment_carried_hash_verification
compact_clears_journal, compact_journal_recovery_roll_forward, compact_orphan_temps_discarded
spend_in_fork_and_compact
./etc/qemu-compact-stress/run.sh --all-phases (native smoke)

- write_tmp_pruned: prune_pos is ordered, so compare against the head of the remaining slice instead of scanning it for every element. Rewriting the PMMR hash/data files was O(elements x prune_count), which could take minutes on low-end hardware after a large prune set accumulated. - init_output_pos_index: binary search for the first header that can contain the first missing output and stop once the last missing entry is indexed, instead of walking every header from genesis to head on each compaction. Helps with mimblewimble#3872.

Segment root validation only checked hashes that participate in reconstructing the segment root. Apply, however, writes every hash in the segment into the local PMMR. A peer could therefore pass a segment with a valid root proof while poisoning intermediate hashes; the corruption only showed up at full state validation as a bad root, with no peer attribution. After the merkle proof succeeds, verify every carried hash is bound to the proof-validated root: either consumed during reconstruction, or a child of an already bound parent that hashes to that parent. Legitimate dense segments from uncompacted peers (pre-mimblewimble#3820 style) still pass when their extra hashes are consistent; only inconsistent or unbound hashes are rejected early as UnverifiableHash. Adds store segment tests covering tampered intermediate hashes and the uncompacted-peer dense format.

A hard kill between replacing hash/data files and flushing the prune list left compacted files paired with a stale prune list, producing wrong shifts and a bad root on restart. Compaction now: 1. Writes pruned hash/data temps and staged prune/leaf bitmaps 2. Fsyncs those temps, then writes a compact journal marker 3. Only then renames staging files into place and clears the journal On PMMRBackend::new, a present journal rolls renames forward; temps without a journal are discarded as abandoned prep. Variable-size size files are rebuilt on re-open when inconsistent with data, matching the existing open path. Adds tests for successful cleanup, roll-forward recovery, and orphan temp discard.

Reproduce slow compaction and hard-kill mid-compact (issue mimblewimble#3872): - compact_crash_stress example: prepare large pruned PMMR, compact with timing, verify root, kill-test that SIGKILLs a child paused at a phase - Test hooks in check_compact (env-gated, no-op in production): ready file + pause at hash_tmp / data_tmp / journal / applied - etc/qemu-compact-stress/run.sh: native, qemu-user (TCG), or Docker linux/amd64 with 0.5 CPU / 512MB (QEMU on arm hosts) - Manual CI workflow for optional long runs

Testnet is usually the easier real-world repro (and often heavier than mainnet even on a desktop): high spend churn and first compact after PIBD rewrites almost the entire hash/data files. Add prune_pct to prepare/kill-test, --testnet-like runner preset (large leaf count + ~92% pruned), and first vs second compact timing so the post-PIBD vs incremental gap is visible.

- Docker --weak: use host platform (not forced amd64 under QEMU), export /usr/local/cargo/bin, source cargo env, fail clearly if cargo missing - Progress logs during prepare/compact so --testnet-like is not silent - Clarify that harness modes never need a manual kill; only real grin --testnet does for optional full-node recovery experiments

ardocrat · 2026-07-16T13:15:32Z

+	/// which only surfaces (without attribution) at full state validation.
+	/// A hash is accepted if it was consumed during reconstruction, or if it
+	/// hashes into an already accepted parent together with its sibling.
+	fn verify_carried_hashes(&self, mut known: HashMap<u64, Hash>) -> Result<(), SegmentError> {


Pruned hashes are still accepted when we are syncing against a node with #3820, so in the end we get Invalid Root before validation

ardocrat · 2026-07-16T13:21:27Z

+/// If a journal is present with staged files, open rolls renames forward and
+/// yields a consistent PMMR (simulates hard kill after journal, before apply).
+#[test]
+fn compact_journal_recovery_roll_forward() {


For some reason test is crashing on Windows

ardocrat · 2026-07-16T13:23:00Z

+/// Staging files without a journal are abandoned prep — discarded on open,
+/// leaving the pre-compact live files intact.
+#[test]
+fn compact_orphan_temps_discarded() {


Test is crashing on Windows

wiesche89

This contains three independent core changes: compaction performance, PIBD validation, and crash recovery with its test harness. Could we split them into three PRs? Each touches a different critical path and would be much easier to review, test, and revert separately.

Could you add before/after timings for both write_tmp_pruned and init_output_pos_index to the PR description? The stress harness is useful, but the expected improvement for #3872 is not recorded yet.

wiesche89 · 2026-07-21T15:49:07Z

+		self.leaf_set.write_to(&leaf_staging)?;
+
+		// --- Phase 2: journal commit point (temps are durable) ---
+		write_compact_journal(&self.data_dir)?;


The journal is the commit point, but the staging directory entries are not synced before it, and the replacements are not synced before the marker is removed. Could we make these directory syncs mandatory and ordered? Otherwise a power loss can leave mixed files with no journal.

wiesche89 · 2026-07-21T15:49:35Z

+		// --- Phase 3: replace live files, then clear journal ---
+		debug!("compact: applying journaled file replacements...");
+		// Release mmaps/handles so renames succeed on all platforms.
+		self.hash_file.release();


If apply or reinit fails after these releases, the backend stays without file handles and callers only log the error or restart the sync loop. Could we complete recovery and reopen it before returning, or propagate this as a fatal node restart condition?

wiesche89 · 2026-07-21T15:50:21Z

+/// Apply any durable compact temps into place and clear the journal.
+/// Safe to call when some temps are already gone (partial prior apply).
+fn apply_compact_temps(data_dir: &Path) -> io::Result<()> {
+	replace_if_present(


The kill phases stop before any replacement or after all of them, but never in between. Could we add kill points after each hash/data/prune/leaf replacement? That is the main partial apply recovery guarantee here.

wiesche89 · 2026-07-21T15:51:50Z

+			let left = known
+				.get(&left_pos0)
+				.copied()
+				.or_else(|| self.get_hash(left_pos0).ok());


get_hash() scans hash_pos linearly and is called for many bound nodes here, making dense peer segments O(n^2). Since this is peer controlled PIBD input, could we index the carried hashes once or use binary search?

wiesche89 · 2026-07-21T15:52:52Z

+///   phase so an external watchdog can SIGKILL the process.
+/// - `GRIN_COMPACT_CRASH_PAUSE_MS`: sleep duration in ms (default 5000 when
+///   pause phase is set).
+fn compact_test_hook(phase: &str) {


Could we gate these test hooks behind a Cargo feature? That keeps env reads and sleeps out of the production compaction path and prevents a stray GRIN_COMPACT_PAUSE_PHASE from pausing a real node.

wiesche89 · 2026-07-21T15:53:42Z

+		);
+	}
+
+	// Hard kill: no graceful shutdown (simulates OOM killer / kill -9 / power loss).


SIGKILL covers process death, but not power loss: the page cache and directory metadata remain. Could we narrow this claim and keep power loss durability separate?

wiesche89 · 2026-07-21T15:54:19Z

+	/// Write the current bitmap to an arbitrary path (used for journaled compact).
+	/// Does not update the in-memory backup; call `mark_flushed` after the write
+	/// is committed via rename.
+	pub fn write_to<P: AsRef<Path>>(&self, path: P) -> io::Result<()> {


This duplicates PruneList::write_to() and both methods are only used by compaction. Could we use one shared crate private bitmap writer instead of expanding both public APIs?

wiesche89 · 2026-07-21T15:54:52Z

+/// file is replaced. Presence means an interrupted compact must roll forward.
+const PMMR_COMPACT_JOURNAL: &str = "pmmr_compact.journal";
+/// Suffix appended to leaf/prune paths for pre-commit compact temps
+/// (`pmmr_prun.bin.compact`). Distinct from `.tmp` used by single-file flushes.


Hash/data compaction still uses .tmp, while prune/leaf uses .compact, so this comment is a little misleading. Could we either use one convention or explain why both suffixes belong to the same journal transaction?

wiesche89 · 2026-07-21T15:55:39Z

+
+	// Keep every Nth leaf so (prune_pct)% are removed. N = 100 / (100 - pct).
+	// e.g. prune_pct=50 → keep every 2nd; prune_pct=90 → keep every 10th.
+	let keep_every = (100u32 / (100 - prune_pct)).max(1);


This only approximates the requested percentage. For example, 60 selects keep_every = 2 and actually prunes about 50%, while the output still reports 60%. Could we calculate or report the actual percentage?

wiesche89 · 2026-07-21T15:56:20Z

 		let total_outputs = outputs_pos.len();
 		let max_height = batch.head()?.height;

+		// outputs_pos is sorted by pos, so binary search for the first header


Could we add an equivalence test for missing entries near genesis and at the head, including consecutive headers with the same output_mmr_size? That would cover the boundaries of the new binary search.

iho added 2 commits July 10, 2026 21:09

iho changed the title ~~Speed up chain compaction~~ Speed up chain compaction and harden PIBD segment validation Jul 10, 2026

iho changed the title ~~Speed up chain compaction and harden PIBD segment validation~~ Speed up chain compaction, harden PIBD validation, crash-safe compact Jul 10, 2026

iho added 3 commits July 10, 2026 21:56

ardocrat reviewed Jul 16, 2026

View reviewed changes

iho force-pushed the compaction-speedups branch from e086d66 to f373dd6 Compare July 18, 2026 05:12

iho mentioned this pull request Jul 18, 2026

Make the sparse PIBD segment optimization (#3820) safe to apply #3915

Open

wiesche89 reviewed Jul 21, 2026

View reviewed changes

Uh oh!

Conversation

iho commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Compaction speed (#3872)

PIBD segment hash validation (PR #3820 fallout)

Crash-safe compaction

QEMU / weak-VPS stress harness

Testing

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wiesche89 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

iho commented Jul 10, 2026 •

edited

Loading