Skip to content

hermes: extend scheduled wake timeout - #346

Merged
mostlydev merged 1 commit into
masterfrom
issue-345-hermes-wake-timeout
Aug 3, 2026
Merged

hermes: extend scheduled wake timeout#346
mostlydev merged 1 commit into
masterfrom
issue-345-hermes-wake-timeout

Conversation

@mostlydev

Copy link
Copy Markdown
Owner

Summary

  • give hermes-exec a dedicated two-minute claw-api wake budget
  • keep the generic 30-second budget explicit for other adapters
  • cover OpenClaw, Hermes, and generic timeout selection in the scheduler test
  • document the fix under Unreleased

This stays in claw-api's adapter-aware wake policy. It does not patch a downstream pod, modify Hermes images, or add cllama behavior.

Verification

  • go test ./cmd/claw-api
  • go vet ./...
  • git diff --check
  • go test ./... passed every package except cmd/claw and internal/build; those two hit the repository's 10-minute timeout while the host Docker CLI was blocked in docker image inspect
  • DOCKER_HOST=unix:///tmp/clawdapus-no-docker.sock go test ./cmd/claw ./internal/build passed both affected unit packages deterministically

Closes #345

@mostlydev

Copy link
Copy Markdown
Owner Author

A fresh concurrency review found a scheduler-level dependency: extending a wake adapter to two minutes is correct, but the current synchronous tick can hold unrelated due invocations behind that wake. #347 now tracks asynchronous, serialized dispatch and overlap protection. Keeping this PR draft; it should not merge until #347 is resolved or incorporated.

@mostlydev

Copy link
Copy Markdown
Owner Author

Dependency update: draft PR #349 now implements the scheduler fairness boundary required before this timeout extension. Its Go and claw-api image checks are green, including focused race coverage for cross-target concurrency, strict per-target FIFO order, cancellation neutrality, and shutdown draining. Keep this PR draft until #349 is reviewed and landed.

@mostlydev

Copy link
Copy Markdown
Owner Author

Independent review (claude, via talking-stick turn 3). Answering the question of whether #348 must land before this PR can honestly close #345.

Determination: this PR does not, by itself, resolve the incident #345 actually reports — but the gap is closed by #349, not by this PR.

#345's reported incident is a manual fire: "A production Hermes schedule fired manually through claw-api ... but claw-api recorded the invocation as wake-error with detail exec timed out." Verified the two client-side deadlines on that path:

  • cmd/claw-api/main.go:59 — local client mode -request-timeout defaults to 10s
  • cmd/claw/api.go:201,215 — outer compose-exec transport bounded at 15s

Both fire well before the 2-minute server budget this PR grants hermes-exec. So on the manual path the client still cuts the request at ~10-15s and the new budget never gets to matter. Raising the server budget alone cannot make a manual Hermes fire return its real outcome.

What each PR actually contributes to #345:

path what fixes it
scheduled (automatic) wake this PR (#346). tick dispatches with the runtime context — no per-request deadline — so execCtx = WithTimeout(ctx, wakeExecTimeout(adapter)) genuinely gets the full 2 minutes. Verified in code.
manual fire, false failure record #349. Neutral cancellation means a client-cut fire returns wake-canceled and is not persisted, so it no longer increments consecutive failures or degrades a healthy schedule. This is the part that stops the operational record from contradicting the runner.
manual fire, operator actually gets the result #348. Still open.

So: #348 is not a merge blocker for this PR, but the Closes #345 claim is only fully honest once #349 has landed too. Recommend merging in order #349#346, and I would soften one clause in the changelog entry. It currently reads as if this change alone prevents a successful Hermes job from being recorded as exec timed out; for the manually-fired case that is #349's neutral-cancellation work. Suggest scoping the sentence to scheduled wakes and pointing at #348 for the manual round-trip.

No objection to the code. Diff is minimal and correct, the test now asserts all three budget classes explicitly (openclaw / hermes / generic default) rather than only the two it covered before, and the changelog touches only ## Unreleased with no pin, badge, or nav changes.

One interaction worth recording: a 2-minute Hermes budget is longer than many cron cadences, which makes #349's overlap coalescing reachable in normal operation. I found that coalesced slots were audit-invisible and pushed a fix to the #349 branch (suppressed_slots / last_suppressed_at in schedule state + a clawdash note) — details in that PR.

mostlydev added a commit that referenced this pull request Aug 3, 2026
Centralizing the longest supported wake budget only helps if something
enforces that it is actually the longest. Nothing did: wakeExecTimeout()
could hand any adapter a budget above schedulepkg.MaxWakeExecTimeout and
the manual fire client, which sizes its request and transport budgets
from that constant, would silently go back to cutting a healthy wake
short -- the exact failure #348 exists to fix. The risk is immediate,
not hypothetical: #346 adds hermes-exec as its own literal.

Adds an invariant test over every adapter resolveWakeAdapter can emit,
plus a nesting assertion so tuning the 5s margins cannot invert the
transport > request > wake ordering. Verified the guard actually bites
by temporarily giving hermes-exec a 3-minute budget, which fails it with
the drift named explicitly.

go test ./... , go vet ./... , git diff --check, and the integration-tag
suite all pass.
@mostlydev
mostlydev marked this pull request as ready for review August 3, 2026 02:04
@mostlydev
mostlydev force-pushed the issue-345-hermes-wake-timeout branch from f42af36 to a2d54e6 Compare August 3, 2026 02:04
@mostlydev
mostlydev merged commit 38e9481 into master Aug 3, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Give Hermes scheduled wakes a runner-sized timeout

1 participant