core, miner, consensus/bor, eth, triedb/pathdb: pipelined state root computation for block import - #2180
core, miner, consensus/bor, eth, triedb/pathdb: pipelined state root computation for block import#2180pratikspatil024 wants to merge 73 commits into
Conversation
…oved the post tx execution buffer time
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.
Tip: disable this comment in your organization's Code Review settings.
Code ReviewFound 6 issues: 4 bugs and 2 security concerns. Bugs
Security Concerns
|
Code ReviewFound 5 issues in miner/worker.go and miner/pipeline.go. Checked for bugs and CLAUDE.md compliance. 1. Bug: writeElapsed always ~0ns (miner/worker.go L1116-L1123) writeElapsed := time.Since(writeStart) is computed immediately after writeStart := time.Now(), before either WriteBlockAndSetHeadPipelined or WriteBlockAndSetHead executes. writeBlockAndSetHeadTimer always records ~0, and workerMgaspsTimer (line 1148) reports inflated MGas/s. Fix: move writeElapsed := time.Since(writeStart) to after the if/else block. 2. Bug: nil pointer dereference (miner/pipeline.go L379-L384) When chainHead is nil, the || short-circuits into the if-body, but chainHead.Number.Uint64() in log.Error dereferences nil and panics. Per CLAUDE.md: No panics in consensus, sync, or block production paths. Fix: split into two if-checks. 3. Bug: unchecked type assertion (miner/pipeline.go L335-L341) borEngine, _ := w.engine.(*bor.Bor) discards the ok boolean. If w.engine is not *bor.Bor, borEngine is nil and borEngine.AssembleBlock(...) panics. The same assertion at line 96 correctly checks ok. Fix: check ok and return early. 4. Bug: goroutine leak on 5 return paths (miner/pipeline.go L293-L345) initialFillDone channel (line 293) goroutine is not drained on return paths at lines 345, 357, 371, 373, 383. Only WaitForSRC error (line 331) and happy path (line 390) drain it. Fix: defer drain after line 293. 5. Bug: trie DB race after SpawnSRCGoroutine (miner/pipeline.go L206-L229) SpawnSRCGoroutine called at line 213 launches a goroutine doing CommitWithUpdate. If StateAtWithFlatDiff fails (line 219) or GetHeader returns nil (line 228), fallbackToSequential does IntermediateRoot inline on the same parent root concurrently. The comments at lines 206-211 identify this as causing missing trie node / layer stale errors but only guard the Prepare() case. Fix: WaitForSRC() before fallbackToSequential, or move spawn after preconditions. |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## develop #2180 +/- ##
===========================================
+ Coverage 54.07% 55.04% +0.96%
===========================================
Files 907 911 +4
Lines 162012 165339 +3327
===========================================
+ Hits 87615 91018 +3403
+ Misses 68984 68889 -95
- Partials 5413 5432 +19
... and 25 files with indirect coverage changes
🚀 New features to boost your workflow:
|
I am okay with the idea of removing the remaining 100ms. We already reduced this buffer from 500ms to 100ms in v2.7.1, and from what we have seen so far, this remaining time looks small enough that removing it seems reasonable. My main concern is not the removal of the 100ms itself. My concern is the cost of pipelining SRC with the next block production. In other words: by doing SRC in parallel with block building, how much do we impact SRC time itself? Do we expect SRC to remain roughly the same, or does it become meaningfully slower because it is now competing with the next block production? That is the part I would like to understand better. I think this is basically a TPS vs finality question:
So I am supportive of the direction, but I think the key question is still: How much TPS do we gain, and how much finality do we lose, if any, by making SRC fully pipelined with block production? If the impact on SRC time is only slight, then the tradeoff is probably clearly worth it. But if SRC time increases materially once it is pipelined with block production, then we should make that tradeoff explicit |
I think SRC will be roughly the same, because the time consuming part, trie nodes prefetching, is already running at the same time with tx execution today, and this PR doesn't change this behavior. |
…r block import Overlap SRC(N) with execution of block N+1 on importing/RPC nodes. After executing block N, defer IntermediateRoot + CommitWithUpdate to a background SRC goroutine and immediately proceed to block N+1 using a FlatDiff overlay for state reads. Cross-call persistence allows the SRC to run across insertChain boundaries. Key changes: - Pipeline path in insertChainWithWitnesses with ValidateStateCheap - FlatDiff overlay in StateAt, StateAtWithReaders, PostExecutionStateAt - Path DB reader chained fallback for concurrent layer flattening - Trie-only reader for SRC witness generation (no flat reader bypass) - WIT handler waits for pipelined witness before returning empty - WitnessReadyEvent for announcing witnesses to stateless peers - PropagateReadsTo in checkAndCommitSpan for witness completeness - Feature gated: --pipeline.enable-import-src
Adds TestPipelinedImportSRC_SelfDestruct to verify that the FlatDiff Destructs check in getStateObject correctly handles self-destructed contracts during pipelined import.
Two fixes for prefetcher errors during pipelined state root computation:
1. Storage root mismatch: FlatDiff accounts had storage roots from block
N's post-state, but the prefetcher's NodeReader was at the committed
parent root (grandparent). Add prefetchRoot field to stateObject that
stores the grandparent's storage root, read from the flat state reader
when loading from FlatDiff. Use it consistently across all prefetcher
interactions.
2. Layer stale during trie node resolution: SRC's cap() flattens diff
layers concurrently with prefetcher trie walks. Add nodeFallback to
reader.Node(), mirroring the existing accountFallback/storageFallback
pattern — retries via the current base disk layer on errSnapshotStale.
A series of fixes for pipelined SRC under EIP-2935/BLOCKHASH aborts and
abort-heavy devnet load:
1. Skip pipeline pre-Rio.
Pre-Rio speculative Prepare walks unsigned speculative headers and can hit
ecrecover failures on zero-seal Extra data. Disable pipelined SRC before
Rio so the miner stays on the safe sequential path there.
2. Move slot waiting fully to Seal and keep abort rebuilds in-slot.
The miner now always builds block bodies early and uses the slot for tx
selection, while Bor holds propagation until the target time in Seal().
Abort-recovery headers carry a miner-local AbortRecovery flag so late
speculative rebuilds stay in-slot instead of getting pushed to the next
slot by minBlockBuildTime.
3. Isolate block-build timeout state per build environment.
Sequential builds and speculative fills previously shared a worker-global
timeout flag, so one build's timer could interrupt another build's tx
selection. Move timeout state onto each environment and make timer cancel
stop the timer without poisoning the build as timed out.
4. Improve speculative fill behavior and fix DAG metadata on refill.
Speculative blocks now take a late refill pass when they are still under
about 75% full by gas and there is at least 300ms left before the slot,
not only when fully empty. Keep tx dependency DAG state on the block
environment across refill passes so multi-pass speculative fills do not
restart dependency indices from zero and drop metadata with
non-sequential transaction index errors.
5. Harden abort recovery and mined-block propagation.
After speculative aborts, requeue normal work through the standard worker
path instead of re-entering commitWork recursively. On the networking
side, mined inline blocks now still announce correctly when witness data
is already cached but the async block write is not yet visible in the DB.
6. Add regression coverage and clean up logs.
Add tests for Bor timing behavior, speculative refill decisions,
per-build interrupt isolation, DAG metadata persistence across refill
passes, cached-witness announcement, and BLOCKHASH(N) abort-flag
behavior. Also remove duplicate EIP-2935 abort logs and fix negative
seal-delay logging so slightly-late blocks no longer print huge wrapped
unsigned delays.
Wires a complete metrics suite for A/B comparing pipelined vs non-pipelined
import and block production on mainnet.
New pipelined metrics (import):
- chain/imports/pipelined/{hit,miss,root_mismatch,enabled}
- chain/imports/witness_ready_end_to_end — apples-to-apples end-to-end timer,
fires in both modes (primary A/B KPI)
New pipelined metrics (build):
- worker/pipelineSpeculativeCommitted, pipelineSRCWait, pipelineSealDuration
- worker/pipelineAnnounceEarlinessMs (signed ms — PIP-66 earliness signal)
- worker/pipelineSpeculativeAborts/{blockhash,src_failed,fallback}
- worker/build_to_announce — producer-side end-to-end, both modes
- worker/pipeline/enabled
Parity wiring for legacy metrics so dashboards work in both modes:
- chain/inserts, account/storage read + hash + update + commit timers,
snapshot/triedb commits, stateCommitTimer, blockBatchWriteTimer,
witnessEncode/DbWrite — emitted from the pipelined branch (main statedb or
SRC goroutine's tmpDB as appropriate)
- worker/writeBlockAndSetHead — emitted from inlineSealAndBroadcast's async
write goroutine
- pipelineAnnounceEarlinessMs and pipelineSpeculativeCommittedCounter also
emitted from resultLoop for the sealBlockViaTaskCh path
Throughput and overlay observability:
- chain/{gas_used_per_block,txs_per_block,mgasps} + chain/witness/size_bytes
- worker/chain/{gas_used_per_block,txs_per_block}
- state/flatdiff/{account_hits,storage_hits} — FlatDiff overlay effectiveness
Metrics that have no clean pipelined semantic (chain/validation, chain/write,
worker/commit, worker/finalizeAndAssemble, worker/intermediateRoot) are left
unemitted in pipelined mode with inline comments documenting the reason and
pointing to the closest pipeline equivalent.
Code ReviewFound 3 issues in
|
Inline Review CommentsSince inline comments could not be posted via the review API, here are the detailed findings with line references: Issue 1 (HIGH): Missing The non-pipelined path (lines 3618-3631) calls Fix: Add the same per-block CLAUDE.md: blockchain-security.md and consensus-security.md Issue 2 (HIGH):
Fix: Acquire CLAUDE.md: security-common.md — "Shared mutable state protected by mutex or atomic operations" Issue 3 (HIGH):
Fix: Replace CLAUDE.md: security-common.md — "Error values checked — never discard errors with _ in security-sensitive paths" |
…ions for diffguard compliance
Decompose large pipelined-src-authored functions into focused helpers so
every function owned by this branch sits under diffguard's 50-line /
complexity-10 limits. Pure structural refactor — no behavior change.
miner/pipeline.go:
- commitSpeculativeWork (599) → orchestrator (35) + specSession struct
with ~18 methods (setupInitial, waitForSRCAndSealBlockN, runOneIteration,
prepareNextIteration, sealCurrentAndAdvance, shiftToNext, etc.)
- inlineSealAndBroadcast (100) → 35 + sealViaPrivateChannel,
rebindReceiptsToSealedBlock, announceInlineSealedBlock
- commitPipelined (59) → 37 + buildSpeculativeReq, spawnSRCForFinalBlock
- sealBlockViaTaskCh (52) → 48 (reuses spawnSRCForFinalBlock)
miner/worker.go:
- fillTransactions (59) → 47 + commitTxMaps
- makeEnv (51) → 38 + resolveStateFor
- updateTxDependencyMetadata (68) → 32 + buildTxDependencyArray
Pre-existing develop functions where pipelined-src had grown the body
are reduced back close to or below their develop size by extracting the
added branches:
- commitWork (67 → 36) via clearPendingWorkOnExit + maybeStartPrefetch
- resultLoop (191 → 124; develop was 123) via emitExecutionMetrics,
emitCommitMetrics, writeTaskBlock, announceTaskBlock
- mainLoop (135 → 120; develop was 116) via handleSpeculativeWork
- buildAndCommitBlock (93 → 83; develop was 80) via submitForSealing
core/state/statedb.go:
- CommitSnapshot (95, complexity 40) → 30 + captureMutation,
captureObjectStorage, captureReadOnlyAccount, captureNonExistentRead
- ApplyFlatDiffForCommit (49, complexity 20) → 16 + applyFlatMutation
- ApplyFlatDiff (36, complexity 11) → 13 + applyFlatAccountOverlay
- TouchAllAddresses (25, complexity 11) → 12 + touchAddressAndStorage,
mutatedStorageKeys
core/blockchain.go:
- SpawnSRCGoroutine (127, complexity 35) → 13 + runSRCCompute,
openSRCStateDB, preloadFlatDiffReads, emitSRCStateDBMetrics,
encodeAndCachePendingWitness
- writeBlockAndSetHeadPipelined (108, complexity 29) → 16 +
writePipelinedBlockBatch, writeBorStateSyncLogs, resolveWriteStatus,
emitPipelinedWriteEvents
- handleImportTrieGC (52, complexity 16) → 21 + capTrieIfDirty,
maybeFlushChosen, dereferenceUpTo
- waitForPipelinedWitness (complexity 11) → 9 + waitForPendingSRCWitness,
pollWitnessCache
core/evm.go:
- SpeculativeGetHashFn (complexity 12) → 17 + newPendingBlockNResolver
core/blockchain.go insertChainWithWitnesses pipelined branch (had grown
+222 lines on top of develop's 452) → +42 via buildPipelineImportOpts,
persistPipelinedImport, collectPrevImportSRCIfAny, emitStateSyncFeed,
runImportAutoCollection, verifyImportSRCRoot, publishImportWitness,
emitPipelinedImportParityMetrics.
core/blockchain.go ProcessBlock pipelined branches (+22 lines) → +4
via pipelineReaderRoot, applyFlatDiffOverlayToAll, validateStateForPipeline.
eth/peer.go:
- doWitnessRequest (pipelined-src pushed from 38 → 65) → 32 +
awaitWitnessResponse extracting the goroutine body
eth/handler_wit.go:
- handleGetWitness (pipelined-src pushed from 70 → 91) → 66 +
resolveWitnessSizes consolidating per-hash size resolution (rawdb +
header-existence DoS guard + SRC cache fallback)
tests/bor/helper.go:
- InitMinerWithPipelinedSRC (65) → 32 + newPipelineTestNode (17),
importValidatorKey (11)
- InitImporterWithPipelinedSRC (64) → 31 (same helpers)
Mutation coverage. Ran diffguard in diff-scoped mode (-base develop
-include-paths <module>) across every module pipelined-src touches and
filled the gaps it surfaced:
- core/state: adds core/state/statedb_pipeline_mutations_test.go with 41
targeted tests that kill 24 of 28 mutation survivors in pipelined-src
FlatDiff code (statedb.go lines 2031-2330, 2492-2499). The 4 remaining
are equivalent mutants — Finalise removes destructed addrs before the
guarded branches can fire (2114, 2163), a zero-length loop produces
the same output with or without the guard (2141), and an empty-slice
map entry is observationally equivalent to a missing entry (2150).
Covers CommitSnapshot and its capture helpers, ApplyFlatDiff +
applyFlatAccountOverlay, ApplyFlatDiffForCommit + applyFlatMutation,
NewWithFlatBase, TouchAllAddresses + touchAddressAndStorage +
mutatedStorageKeys, WasStorageSlotRead, and PropagateReadsTo — 14 of
15 functions at 100% line coverage (captureReadOnlyAccount at 90.9%).
- core/stateless: extends witness_test.go with 3 tests targeting
ValidateWitnessPreState's expectedBlock guard (ParentHash and Number
checks that defend against a malicious peer substituting a witness
for a different block / fork). Previous tests all passed nil for
expectedBlock, leaving the entire anti-forgery branch uncovered.
- eth/filters: adds TestResolveBlockNumForRangeCheck and
TestCheckBlockRangeLimit (16 subcases) to api_test.go covering the
RPC range-limit DoS guard at the unit level (sentinel resolution,
span-at-limit boundary, sum-vs-span distinction). Extends
TestInvalidGetRangeLogsRequest in filter_system_test.go to also
exercise GetBorBlockLogs with an inverted range — previously only
GetLogs was covered.
Per-module mutation scores after this coverage: miner 96%, consensus/bor
100% (41/41), core 100% (447/447), core/state 86% (24/28 equivalent),
core/stateless 100% (8/8), core/txpool 100% (12/12), tests/bor 100%
(43/43), triedb/pathdb 100% (20/20). eth at 53% — remaining survivors
are in auto-generated gen_config.go boilerplate (36), P2P dispatcher
cancel-channel plumbing awaitWitnessResponse goroutine cleanup
(3); documented as accepted gaps requiring complex mock infrastructure
for diminishing security return.
Remaining diffguard violations in miner and core are pre-existing
develop functions (commitTransactions, insertChainWithWitnesses,
newWorkLoop, NewBlockChain, ProcessBlock, writeBlockWithState, etc.)
that were over threshold on develop before pipelined-src. Their
pipelined-src deltas are now small (+1 to +42 lines) and out of scope
for this PR.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Renames (reviewer nits):
- PostExecutionStateAt → PostExecState (BlockChain + txpool/legacypool/
blobpool interfaces + test mocks).
- ResetSpeculativeState → SetSpeculativeState, SpeculativeResetter →
SpeculativeSetter (the method overwrites, it doesn't revert).
Dedup between writeBlockAndSetHead and writeBlockAndSetHeadPipelined.
Both paths now share resolvePostWriteStatus(block, stateless) for
fork-choice + reorg (stateless flag preserves the errInvalidNewChain
escape for fast-forward sync), emitPostWriteEvents for the feed sends,
and writeBorStateSyncLogs for the pre-Madhugiri bor receipt. ~80 lines
of duplicated fork-choice + event logic removed; writeBlockAndSetHead
drops from ~70 to ~11 lines. Batch bodies intentionally not merged —
witness source (statedb.Witness() vs pre-encoded bytes) and trie-commit
timing genuinely differ.
Miner coinbase unification: extracted resolveCoinbase(blockNumber,
fallback) used by makeHeader (fallback=genParams.coinbase) and the
speculative header builders (fallback=etherbase()). Divergence between
the speculative and real header would cause a state root mismatch, so
single-sourcing this is security-meaningful. Rest of buildInitialSpecHeader
kept separate from makeHeader (placeholder parent, deterministic bor
period timestamp, static GasCeil, no engine.Prepare); comment documents
why unifying further would hurt readability.
Pipelined import correctness fixes (core/blockchain.go):
1. persistPipelinedImport now runs the Heimdall milestone/checkpoint
ValidateReorg guard before writeBlockAndSetHeadPipelined — mirrors
the non-pipelined path. Without it, a milestone whitelisted during
block execution could be bypassed.
2. verifyImportSRCRoot wraps its writeHeadBlock revert in chainmu
TryLock/Unlock. The call ran in the auto-collection goroutine
without the mutex, racing any concurrent InsertChain on head state.
Skips + warns if chainmu is closed (shutdown).
3. flushPendingImportSRC error in insertChain's ProcessBlock error
path no longer discarded with `_`; logged like the other two call
sites.
Linter: dropped two `tc := tc` loop-var copies in eth/filters/api_test.go
(copyloopvar, redundant since Go 1.22).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Re: the review-level HasState()/rewindHead analysis from claude[bot] — confirming the ruled-out reasoning is correct: pendingImportHeadHash/pendingImportHeadStart and pendingImportSRC are process-local and zero-valued on a fresh process, so crash-recovery rewindHashHead/rewindPathHead at startup never see pipelined-pending state as available; on a live SetHead the grace window only covers a root whose SRC commit is already in flight and bounded. No code change needed. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 73 out of 74 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (2)
core/stateless/witness.go:76
contextHeader.Number.Uint64()-1underflows when the witness context header is block 0, causing a parent lookup at MaxUint64 (and misleading error messages). Add an explicitNumber==0guard and computeparentNumonce for both lookup and error reporting.
This issue also appears on line 109 of the same file.
parentHeader := headerReader.GetHeader(contextHeader.ParentHash, contextHeader.Number.Uint64()-1)
if parentHeader == nil {
return fmt.Errorf("parent block header not found: parentHash=%x, parentNumber=%d",
contextHeader.ParentHash, contextHeader.Number.Uint64()-1)
}
core/stateless/witness.go:113
- In
NewWitness,context.Number.Uint64()-1can underflow for block 0 and will also panic ifcontext/context.Numberis nil (the latter can occur if callers pass an empty header whilechain != nil). Add validation and use a computedparentNumfor both the lookup and error path.
parent := chain.GetHeader(context.ParentHash, context.Number.Uint64()-1)
if parent == nil {
return nil, errors.New("failed to retrieve parent header")
}
headers = append(headers, parent)
1b34b2a to
94bfc99
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 74 out of 75 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (2)
core/stateless/witness.go:72
- ValidateWitnessPreState dereferences expectedBlock.Number and contextHeader.Number without nil/zero guards. Since this is an exported function, a caller passing an incomplete header (Number == nil) or a genesis/zero-number context will panic or underflow in the parent lookup (Number-1). Add defensive checks and return a descriptive error instead of panicking.
// Verify the witness is for the expected block — a malicious peer could
// craft a witness with a different ParentHash to bypass the pre-state check.
if expectedBlock != nil {
if contextHeader.ParentHash != expectedBlock.ParentHash {
return fmt.Errorf("witness ParentHash mismatch: witness=%x, expected=%x, blockNumber=%d",
contextHeader.ParentHash, expectedBlock.ParentHash, expectedBlock.Number.Uint64())
}
if contextHeader.Number.Uint64() != expectedBlock.Number.Uint64() {
return fmt.Errorf("witness block number mismatch: witness=%d, expected=%d",
contextHeader.Number.Uint64(), expectedBlock.Number.Uint64())
}
}
eth/handler.go:629
- witnessReadyCh is subscribed to an event.Feed, and Feed.Send blocks until all subscriber channels accept the value (see event/feed.go:71-72). With a buffer of 10, a burst of WitnessReadyEvent during fast catch-up can backpressure the SRC goroutine at bc.witnessReadyFeed.Send, potentially reducing import throughput. Consider using a larger buffer (similar to txChanSize) to avoid stalling the pipeline on transient handler/peer slowness.
94bfc99 to
a0d0a0b
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 75 out of 76 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (3)
eth/handler.go:629
- The witness-ready event subscription uses a channel buffer of 10.
publishImportWitnessemitswitnessReadyFeed.Send(...)from the SRC auto-collection goroutine, andevent.Feed.Sendis synchronous; if this channel fills, it can backpressure that goroutine (delayingcollectedChclose and potentially timing outWaitForPipelinedStateCommit/witness waits). Use a larger buffer consistent with other hot-path event channels to avoid stalling pipelined import under bursty witness production.
tests/bor/pipelined_rpc_test.go:81 - This integration test creates two nodes backed by temporary datadirs but never removes those directories. Over many integration runs (or retries), this can leak disk space in CI runners. Since
node.NodeexposesDataDir(), addt.Cleanupto remove the temp dirs after the stacks are closed.
core/state/database.go:207 - The new
TrieOnlyReaderhelper delegates tonewTrieReader, which can open either MPT or Verkle depending on the underlying triedb. The comment currently states it is "only the trie (MPT)" and that it is required for witness building, which is MPT-specific. Please adjust the comment so it matches the actual behavior and doesn’t mislead Verkle callers.
// TrieOnlyReader returns a state reader that uses only the trie (MPT), skipping
// flat/snapshot readers. This ensures all account and storage reads walk the trie,
// which is required for witness building — the witness captures trie nodes during
// the walk. Without this, flat readers short-circuit the trie and proof paths are
// never captured.
a0d0a0b to
5d40179
Compare
5d40179 to
1fdd4dd
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 79 out of 80 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
eth/handler.go:627
witnessReadyChis subscribed tocore.BlockChain.witnessReadyFeed.event.Feed.Sendcan block when a subscriber channel is full (slow subscribers are not dropped), so a buffer of 10 risks stalling the pipelined SRC collection path under high import throughput (or slow peer iteration), extending the “head advanced, witness/state not committed yet” window. Consider using a larger buffer to keep witness-ready announcements from backpressuring import/SRC.
1fdd4dd to
11636e8
Compare
Description
Overlaps state-root computation (SRC) of block N with transaction execution of block N+1 during block import. Execution of N+1 opens state at the last committed trie root plus an in-memory
FlatDiffoverlay of block N's mutations, while a background SRC goroutine commits block N and computes its root. Disabled by default — enabled with[pipeline] enable-import-src = true.Built on top of the delayed SRC PoC: instead of just deferring SRC, it pipelines SRC with the next block's work.
Measured impact
Mainnet full node (n2d-standard-16, path scheme, BlockSTM enforce), ~64k-block catch-up windows, block-weight and traffic-character controlled across adjacent weekday ranges:
producewitnessesproducewitnessesPer-block overhead beyond raw execution dropped from ~45–50ms to ~18ms. Zero
root_mismatch, pipeline fallbacks, or execution errors across the benchmark campaign (~500k blocks imported under the pipeline).How it works — import
When importing block N (pipeline active):
ValidateStateCheap(gas, bloom, receipt root — noIntermediateRoot)FlatDiffviaCommitSnapshot(~1ms, no trie hashing)StateAt/RPC reads (eth_call,eth_estimateGas, pending reads) are correct during the pipeline windowTwo supporting optimizations ship with the pipeline:
triedb/pathdb/lookup_nodes.go): a reference-counted(owner, path, hash) → blobmap over live diff layers.reader.Noderesolves in one probe instead of walking up to 128 diff layers per read (the walk measured ~44% of process CPU during catch-up). The hash is part of the key, so a hit is correct by construction; a definitive miss reads the disk layer directly. This index is always on (not gated by the pipeline flag) — it is a read-path cache whose misses fall through to the existing walk, adding ~250MB steady-state at 128 diff layers.core/state/warm_snapshot.go,pipeline.warm-snapshot, default on): when witnesses are produced, the execution-side trie prefetcher is detached and handed to SRC, which builds an immutable hash-verified snapshot of the loaded trie nodes and consults it before pathdb. No effect when witnesses are off.Review-sensitive properties
ValidateStateCheapgates insertion; the full root check happens asynchronously in the SRC goroutine. A mismatch fireschain/imports/pipelined/root_mismatch(a hard alarm that must stay 0) and errors the pipeline. Sync compensates:hasPendingPipelinedHeadStatekeeps the node in full-sync while a head-state commit is in flight.WitnessReadyEventnow push-announces witness availability to stateless peers (replacing a 10s poll);GetWitnessserving waits for in-flight SRC (bounded, with a header-existence DoS guard) and reads uncached so peer traffic can't evict import-critical cache entries.NewTrieOnly) so every read walks the MPT and proof paths land in the witness; parity tests assert proof-node-set equality and stateless replay (ProcessBlockWithWitnesses) across all configurations.PropagateReadsToincheckAndCommitSpancaptures the validator-contract proof nodes read via a copied statedb; EIP-2935 accounts are touched into the read set.bc.wg; pending SRC is flushed on reorg, gap, or error.BlockChainVersionunchanged); nothing forces a resync.Production-side (miner) pipelining: landed but disabled
miner/pipeline.gocontains the speculative-sealing counterpart (FlatDiff extraction afterFinalizeForPipeline, background SRC, speculative N+1 build, async chain write). It is hard-disabled:isPipelineEligiblereturnsfalseunconditionally and no config exposes it (theworker/pipeline/enabledgauge is always 0). Pre-Rio seal-recovery interaction makes speculativePreparefail; re-enablement is future work and the gating logic is preserved in comments. Reviewers can treat the miner path as dormant code with test coverage.Config
Notable negative results baked into the design (so nobody re-litigates them): the exec-side trie prefetcher and the speculative block prefetcher are load-bearing via shared-cache warming — disabling either regresses 25–75% — so neither is configurable (
pipeline.exec-prefetchexisted briefly on the feature branch and was removed).Executed tests
-race): pipelined vs sequential roots/hashes, witness proof-node-set parity + stateless replay, FlatDiff mutation/overlay suites, self-destruct integration, pathdb node-index unit tests (fork refcounting, deletion markers, definitive-miss semantics),tests/borintegration.core/pipelined_window_test.go, SRC goroutine held open via test hook) pins the overlay/trie semantics of the "head advanced, root not yet committed" interval; an end-to-end battery (tests/bor/pipelined_rpc_test.go) drives 12 read methods against a live-syncing importer at pinned heights andlatest, checks sync-time vs settled response identity, and verifies full per-height parity vs a non-pipelined BP including cryptographic account-proof verification. Handlers that need committed trie nodes (eth_getProof,debug_storageRangeAt) gate onWaitForPipelinedStateCommitinstead of erroring transiently inside the window.Quality gates — declared deviations
Three CI gates fail for structural reasons; each is intentional and listed here rather than suppressed:
insertChainWithWitnesses,IntermediateRoot,getStateObject, andupdateTrieexceed thresholds. All four are legacy hot-path giants that this PR extends in place; refactoring them is high-risk consensus-path churn that belongs in a dedicated cleanup, not a feature PR. New-code violations were extracted below thresholds (applyFlatMutationFast,runSRCCompute,FinalizeForPipeline, witness-collection loops deduped intoaddObjectWitness).nodeWalkis a rename of pre-existing upstream logic. The reportedcore -> coredependency cycle is a tool artifact (self-cycle).addObjectWitness,commitSprintWorkreuse);ValidateStateCheapintentionally mirrorsValidateState's shape.tests/bor, kurtosis). The 90% patch bar on a 12.7k-line PR spanning hot paths is not attainable without duplicating e2e coverage as unit tests.Rollout notes
Backwards-compatible; off by default; no coordinated upgrade required. Not consensus-affecting: the pipeline computes the identical root on the identical commit path, one block later in wall-clock, and aborts on mismatch. Operator-facing changes when enabled: witness-ready latency trails the block by one SRC cycle (~210ms end-to-end vs ~143ms inline). Related: #2312 implements an alternative trie-node index on develop; this PR lands first and #2312 rebases.