Skip to content

Quarantine durability: persist causes, survive restarts, and support re-driving quarantined frames #161

Description

@MaxGhenis

Follow-up from the 2026-08-01 quarantine incident (fixed by #160). Three gaps made it needlessly hard to diagnose and impossible to heal retroactively:

  1. Causes are ephemeral. markQuarantined logs the cause at warn level and bumps an in-memory counter. The backend's stdout reaches os_log at info level via BackendManager, and info entries live in a short memory ring — by the time the sentinel flagged quarantined: 3, the cause lines were unrecoverable. Diagnosis required replaying the inbox frames through the worker offline in a package test. Persist quarantine records (inbox_id, account, codec, cause, occurred_at) in a table, surfaced via /api/status (last N causes) and/or a CLI subcommand.

  2. Counters reset on restart. v2_ingest counters are in-memory, so every app relaunch zeroes quarantined and an external monitor loses the signal. A persisted count (or the table above) keeps the evidence.

  3. Quarantined frames are terminal. They're marked processed and never re-driven, even though the full payload persists in the inbox table. After a decoder/projector fix deploys (like Dedupe duplicated conversation participants instead of quarantining the snapshot #160), a requeue quarantined maintenance command could clear processed_at_ms for quarantined rows (or replay them directly) so the dropped events project instead of waiting for the platform to happen to resend the same state.

The #160 diagnosis recipe (replay inbox frames via processRecordResult against a store copy) is documented in the PR and works today, but it shouldn't be the only path.

🤖 Generated with Claude Code

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions