No authoritative pause: two scheduler surfaces drift, one-shot fire re-arms cron, paused agent burned 610k tokens in 55 min

[mostlydev]

Summary

An operator-paused agent burned ~610k prompt tokens in 55 minutes because "paused" is not one state — it is two independent scheduler surfaces plus an LLM-egress path, and no single handle closes all three. We hit every edge of this in one day, on real OpenAI billing. Related to #356 (manual fire semantics) but broader: this is a request for one authoritative pause handle.

What happened (emmett-pod, hermes driver, 2026-08-03)

1. Pause is two surfaces that drift. claw api schedule pause <id> set the Claw invocation to paused: true — but the hermes native cron kept firing at */2. An agent the operator believed paused churned for hours. (First incident, ~03:00Z.)
2. claw api schedule fire on a paused invocation refuses (records skipped / paused-by-operator). Reasonable — but it pushed us to the workaround below. There is no "fire once while staying paused" affordance for controlled/supervised wakes.
3. hermes cron run <id> re-arms the schedule as a side effect. Its own output says it: Triggered job … Next run: 13:42:16 … It will run on the next scheduler tick. We used it for three supervised proof wakes; it silently resurrected the */2 cadence behind the still-paused Claw invocation. Result: 51 unattended LLM turns / 609,846 prompt tokens between 13:50Z and 14:45Z, all no-ops, all billed. (Second incident.)
4. claw up rematerialization resets state adjacent to pause: it re-mints the invocation id and (separately observed) resets the cllama image; whether schedule pause state survives rematerialization is not documented or verifiable in one place.

The application-level design (server-computed actionability verdicts) meant zero unwanted actions occurred — every churned wake was a correct silent no-op. The spend is the damage: paused-but-churning is invisible until the bill.

What we ask for

1. One authoritative pause. claw api schedule pause (or a new claw pause <agent>) must pause the driver-native scheduler too — for hermes, that means the in-container cron. A pause that governs only claw's own evaluator is a UI, not a control.
2. One-shot fire that does not re-arm. A supervised single wake (claw api schedule fire --once, working even while paused, or a hermes cron run --no-rearm) is exactly what operator-gated acceptance testing needs. Today the only working one-shot (hermes cron run) silently resurrects the schedule.
3. Pause state must survive claw up (or the regenerated pod must come up paused when it was paused before), and claw ps/claw health should surface a per-agent line: schedule: paused (claw) / paused (driver-native) / llm-egress: open.
4. Optional but the real backstop: a cllama-level agent kill-switch / budget ceiling. cllama already owns the only provider key (credential starvation) and already records reported_cost_usd per turn. A per-agent paused: true or daily token/cost ceiling in cllama config would make runaway-schedule spend structurally impossible regardless of scheduler bugs: wake all you like, the proxy refuses upstream. This is the only layer where "paused" can be made fail-closed against every future scheduler surface.

Evidence

• Claw state during churn: .claw-governance/schedule-state.json paused: true, last_detail: paused-by-operator while hermes fired */2.
• Re-arm output: hermes cron run 7172e48faf83 → Next run: 2026-08-03T13:42:16 … will run on the next scheduler tick, immediately after hermes cron list showed No scheduled jobs.
• Session history emmett/history.jsonl: 51 records 13:50Z–14:45Z, 609,846 prompt tokens, zero tool calls in each.