Skip to content

Real trace/eval ingestion from public-safe provenance (replace synthetic fixtures) #2

Description

@AtlantisPleb

Why

The first slice optimizes over tiny synthetic in-repo fixtures (src/mutalisk/eval_set.py). Component 2 of docs/ARCHITECTURE.md is real trace + eval ingestion: pull public-safe executed traces and eval results (provenance refs only; no raw prompts/customer data) as the optimizer training/validation sets.

Scope

  • Define an ingestion interface that loads eval/train datasets from a public-safe source (D1/R2 provenance refs) into the Example/dataset shape the optimizers consume.
  • Enforce redaction/public-safety at the boundary: no customer prompts, no hidden reasoning traces, no wallet material, no secrets in traces or candidates.
  • Carry real eval_evidence_refs / trace_provenance_refs through to emitted candidates (today these point at the in-repo fixtures).
  • Keep ingestion offline/batch; no online/hot-path reads.

Invariants (AGENTS.md)

  • Fail closed when provenance or metric evidence is missing.
  • Do not put production data, customer prompts, or wallet material into traces or candidates.

Done when

  • Optimizers can run over an ingested public-safe dataset (with the synthetic fixtures retained as a smoke/default), provenance refs flow into candidates, and tests cover the redaction/fail-closed boundary.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions