Skip to content

fix(devshard): let a node replay its own pre-floor diffs on startup - #1595

Open
qdanik wants to merge 2 commits into
gonka-ai:gateway-v4from
qdanik:qd/fix-max-tokens-recovery
Open

fix(devshard): let a node replay its own pre-floor diffs on startup#1595
qdanik wants to merge 2 commits into
gonka-ai:gateway-v4from
qdanik:qd/fix-max-tokens-recovery

Conversation

@qdanik

@qdanik qdanik commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Problem

enforce inference tokens floor (#1391) added a MinTokensFloor check to applyStartInference. That check also runs on replay, so a node that served inferences before the floor landed cannot restart — it re-runs a rule written after its own diffs were made:

replay nonce 12: max_tokens below min_tokens floor: max_tokens 1, floor 64

Refusing the diff undoes nothing. It is already part of a recorded state root; the refusal only keeps the node down.

Fix

Separate authoring new work from replaying work already accepted.

  • state.ApplyPersisted sets replayingPersisted under the same mutex as the apply and clears it via defer
  • the floor check is skipped only on that path
  • both replay call sites switch to it: user/recover.go (gateway) and cmd/devshardd/session/manager.go (host daemon)

Exactly one check is relaxed, and only the one guarding creation of new work. Live traffic still has to meet the floor.

Tests

Test Package Covers
TestApplyPersisted_ReplaysAStartWrittenBeforeTheFloor state the policy split, and that the relaxation does not outlive the call
TestRecoverSession_ReplaysADiffWrittenBeforeTheMaxTokensFloor user the gateway's recovery path, from a seeded sub-floor diff
TestRecoverSessions_ReplaysADiffWrittenBeforeTheMaxTokensFloor cmd/devshardd/session the host daemon's recovery path

Each fixture asserts ApplyLocal still rejects the diff before seeding it, so the test cannot silently stop reproducing the failure.

Mutation-tested — 4/4 killed, each verified to compile first: floor ignores the replay flag; flag never cleared; each replay site reverted to ApplyLocal.

state, user and cmd/devshardd/session pass in full.

…m, so a node whose history predates the max_tokens floor can start
@qdanik
qdanik requested a review from a-kuprin August 14, 2026 14:08
Comment thread devshard/state/machine.go
Comment thread devshard/state/machine.go Outdated

// ApplyPersisted replays a diff this node already accepted. A diff is part of a recorded state root,
// so a rule that tightened since it was written cannot undo it by refusing it, only fail to start.
func (sm *StateMachine) ApplyPersisted(nonce uint64, txs []*types.DevshardTx) ([]byte, error) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would better name it ApplyLocalPersisted to keep some mental link with ApplyLocal

and keep the comment from ApplyLocal:

// ApplyLocal applies txs without signature verification. Used by the user
// to compute the post_state_root before signing the diff.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants