Skip to content

Prevent slow scheduled wakes from blocking unrelated invocations #347

Description

@mostlydev

Problem

cmd/claw-api/scheduler.go evaluates every due invocation synchronously inside one minute tick. A wake adapter may legitimately take up to its bounded runner-specific timeout (currently two minutes for OpenClaw, and proposed for Hermes in #346). While one wake is running:

  • later invocations due in the same tick do not start
  • the scheduler timer is not reset
  • invocations due on later minute boundaries can also start late

This couples unrelated agents and targets. Extending one adapter's valid wake budget therefore increases fleet-level scheduling latency even though the timeout itself is correct.

Current code path

  • scheduler.Run calls tick synchronously and resets its minute timer only after tick returns.
  • scheduler.tick loops through due entries and calls dispatch inline.
  • dispatchWithOptions blocks in shared.ExecInContainer until completion or timeout.
  • FireNow can race a scheduled dispatch for the same invocation because no in-flight guard is shared between the automatic and manual paths.

Required behavior

  1. A slow wake for one target must not delay due wakes for unrelated targets.
  2. The minute scheduler loop must remain responsive while a bounded wake is in flight.
  3. The same invocation must never execute twice concurrently, including scheduled versus manual fire.
  4. Multiple due invocations for one runner target must not race the runner's native scheduler or state files.
  5. Schedule state and failure/degraded accounting must remain ordered and auditable for each fire slot.
  6. Work must remain in claw-api scheduler transport/state. Do not add downstream wrappers, runner prompt behavior, or cllama logic.

Suggested shape

Use asynchronous dispatch with bounded per-invocation in-flight tracking and target-level serialization. Keep at most one queued/running dispatch per invocation so repeated minute ticks cannot create an unbounded backlog. Preserve the existing fire-slot timestamp and next-fire semantics unless a documented decision explicitly changes missed-fire handling.

Verification

  • focused tests prove unrelated targets start without waiting for a blocked wake
  • focused tests prove same-target dispatch is serialized
  • focused tests prove a second scheduled/manual fire of the same invocation is rejected or deferred while in flight
  • schedule state remains correct after completion/failure
  • go test ./cmd/claw-api
  • go test -race ./cmd/claw-api
  • go test ./...
  • go vet ./...
  • git diff --check

Relationship

PR #346 should remain draft and depend on this issue's correction before extending the Hermes wake budget.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions