E10: Observability and SRE for Sith itself

[gnanirahulnutakki]

Goal. make the hub — the crown jewel — observable and operable: metrics, tracing, and structured logs about Sith's own behavior, SLOs with error budgets, and the hardening a fleet-wide control plane demands.

Phase / depends. the hardening posture is day-one; the surfaces mature P1 → P3. Depends on: E9 (deployment) and touches every other epic. This epic observes Sith itself — it does not store other systems' telemetry (that would be a telemetry lake, out of scope). The hub is the highest-value target and largest blast radius in the estate, so its own operability and hardening are first-class.

Features

• F10.1 — metrics
• F10.2 — distributed tracing
• F10.3 — structured logging
• F10.4 — SLOs and alerting
• F10.5 — crown-jewel hardening

Exit criteria

• Sith exposes metrics, traces, and structured sanitized logs about its own behavior, correlated by intent id, with no secret leakage and no other-system series retention.
• SLOs with error budgets cover read freshness, dispatch success, and PDP latency; alerts cover reliability and security conditions.
• Crown-jewel hardening is in place: KMS/HSM key custody, RLS + least-privilege hub identity, signed images + SBOM, pinned/verified addons.

---

Source: docs/EPICS.md, docs/SITH-NOTION.md · part of the Sith implementation backlog (see the master roadmap issue).

[gnanirahulnutakki]

P1 policy observability slice #115 is complete via PR #116 (merge 85a4958). The hub PEP now has a sanitized structured local audit sink: it validates normalized events before emission, logs only timestamp/workspace/actor/role/action/closed-verb/verdict/reason code, and keeps raw arguments, selector/digest, credentials, targets, connector data, endpoints, persistence, telemetry, and metrics out of scope. Exact post-merge CI—including race, forced PostgreSQL RLS isolation, release/SBOM, and two-kind-cluster fan-out—is green: https://github.com/ArdurAI/sith/actions/runs/29204163679

[gnanirahulnutakki]

P1 F10.1 metrics foundation is delivered through #119 / PR #120 (dev merge cdb8460). Sith now has a dedicated non-global Prometheus registry and embeddable pull-only handler; PEP and federated snapshot paths emit only fixed outcome vocabularies with no workspace, actor, cluster, resource, selector, credential, or raw-error labels. The slice adds no listener, exporter, telemetry persistence, or external-system series.

Exact post-merge CI 29219341754 passed all gates, including forced RLS isolation, reproducible release/SBOM/formula checks, and real two-cluster fan-out. E10 remains open for future hub runtime wiring, readiness/DB signals, tracing, SLOs/alerts, and crown-jewel hardening.

[gnanirahulnutakki]

P1 F10.2a is complete via #137 / PR #138 (dev merge 1ecfb56). The hub now mints a local opaque trace ID only after signed scope derivation, strips caller-provided trace/correlation headers, correlates PEP audit events with bounded local PEP/snapshot stage records, and carries the same context into the current OCM snapshot transport. Runtime records only trace ID, fixed stage/outcome, and bounded duration; there is no OTel SDK, external exporter, listener, queue, persistence, trace store, or generic attributes.

Exact post-merge CI 29335900057 and CodeQL 29335899220 passed, following zero-finding independent CodeRabbit review and local race/RLS/release/two-cluster gates.

E10 remains open. Full F10.2 still needs E4/E6 typed intent and ledger schemas before it can bind trace_id == intent_id across proposal, PDP, dispatch, and spoke actions. This slice deliberately does not claim that future protocol.

[gnanirahulnutakki]

E10 F10.3a landed in dev via PR #141 and merge commit ba102f4.

It adds a closed, privacy-preserving hub authentication-refusal signal: one local structured WARN record with fixed hub-auth and refused fields. Refusals retain one uniform HTTP 401 response, and logs carry no token/header text, request metadata, workspace, principal, verifier error, or pre-auth correlation ID.

Validation: PR CI 29339209585 and exact post-merge CI 29339882829 passed; post-merge CodeQL 29339878929 passed. GitHub security queues are 0 / 0 / 0. Follow-up #140 separately tracks bounded nonblocking delivery with explicit worker lifecycle.

[gnanirahulnutakki]

E10 F10.1b is complete: PR #178 merged as f5a2c00, and exact post-merge dev CI run 29397856646 passed. It composes the bounded-label operator registry into an opt-in exact-loopback-only Hub endpoint without tenant routing, Service, ingress, or remote scraping. This unblocks the bounded audit-log drop counter in #140.

[gnanirahulnutakki]

E10 F10.3b is complete: #140 closed after PR #179 merged to dev as 977372b. Exact post-merge dev CI 29402917099 passed all gates in 9m28s, including the real two-cluster fan-out test. The Hub now has bounded process-supervised authentication-refusal delivery with opt-in loopback-only loss observation and a regression test for disabled metrics binding.