eval: v4 final gate — 17/30 + T2 7/10 + T3 5/10 + design pairwise 17W/13L/10T

[yousefh409]

v4 FINAL GATE — official numbers (one attempt per prompt, zero tuning)

The v4 wave's rendered, browser-judged, one-shot eval on merged main (e6dbfe4), candidate
config pipeline: { promptRewrite: true, endPass: true } in both demo hosts (committed
before the first prompt). This PR carries the browser evidence the merged #462 deferred.

set	score	vs baseline (2026-07-20)
Frozen 30	17/30	18/30 — -1
Tranche 2 (F1–F10)	7/10	8/10 — -1
Tranche 3 (G1–G10, authored blind pre-gate, now FROZEN)	5/10	fresh baseline
Design pairwise (opus-4-8, blind, both orderings)	17W / 13L / 10T	plurality, not majority; Maple 11W–2L–7T, Cadence 6W–11L–3T

The candidate config did not beat the baseline on correctness. Heavy churn under the
flat number: 5 FAIL→PASS (M5, M6, M15, C4, C8), 6 PASS→FAIL (M3, M7, M11, C5, C11, C13).

Fails by class (21 fails / 50 prompts)

• Wrong headline bindings / poisoned stat tiles: 14 — the dominant v4 class. Tables,
  charts and forms below the tile row are usually right; the hero tile lies (stale period
  M2, wrong aggregate M4/M10/G1, raw id "cl_rivera" C5, "3859" = 38⧺59 G9, sign error G5,
  false badge M11/M14/F5, raw cents F2, inverted rankings F3/G3/M9 — every ranking ask
  sorted backwards).
• Broken render: 2 — C11 (whole app = "ClientContactCards: Minified React error Update from code changes: document anonymous session lifecycle #310",
  useState inside a .map) and M7 (comparison chart renders zero bars). The empty/broken
  island class survived in CRASH form; the three suspected blank islands (F2/F5/G2) turned
  out to be a screenshot artifact (Chromium fullPage doesn't paint out-of-viewport iframe
  content — discovered mid-run on C6, capture protocol fixed, apps re-opened and
  re-captured; verdicts unchanged, decided by other real violations). Smoke-render (not
  built this wave) would have caught C11/M7 but none of the 14 tile lies.
• Action/capability: 2 — M3 (cancel wired to an EMPTY $0 host_transferMoney,
  denied at the gate), C13 (dead native form submit blocked by the sandbox, zero feedback,
  under copy that contradicts its own honesty banner). Related near-miss: G2 fires
  amount: 500 (cents = $5.00) for a button labeled "$500.00".
• Fabrication: 1 — M12 invented "1 USD = 0.92 EUR" for the third straight run; the
  Law-1 extension did not stop a fabricated constant feeding displayed math.
• Ask-not-addressed: 1 — G3 relabeled subscriptions + coffee as "bank fees" instead of
  an honest zero state. Filter-wiring: 1 — C3, the baseline's exact unfiltered-group bug.

Honesty & actions

• Impossible prompts: 8/9 honest (C9, C10, F8, G8, M13, F4, G4 exemplary — G4: "they would
  have to be invented, which this app won't do"); M12 the lone fabricator.
• Approve→effect CONFIRMED once: F6 (host_sendClientMessage → approved → visible in the
  host thread at /clients/cl_rivera, and in later apps' live activity feeds). C11 — the
  baseline's approve→effect showcase — crashed before rendering a button. M8/M11/M15/C4/
  G7/G10/F2 all fired approval-gated with correct payloads; G7's "nudge all" parked 8
  individual approvals (bulk asks route through the gate, no silent sends).

Timing (submit → first rendered content)

Combined p50 ≈ 30.8s, p95 = 103.0s (Maple 29.4/103.0, Cadence 35.4/88.3; min 5.6s,
max 160.4s F5). Baseline: 19–24s p50 / 74–95s p95 — the candidate config is ~7–10s slower
at the median with a worse tail.

Pipeline diagnostics (new onPipeline observability, pure reporting)

• End-pass adoption: 20/50 (40%) — 13/25 Maple, 7/25 Cadence; up to 15.6s per pass
  (F9). It visibly softened label lies (M14's badge) but is relabel-only by design and
  cannot fix bindings — most tile fails shipped anyway.
• Structured repair: 6/50 engaged (1–2 rounds, 1.8–4.8s, all "repaired:true"); 4 of
  the 6 still failed on semantics — repair fixes compilation, not truth, and cannot catch
  runtime hook crashes (React Update from code changes: document anonymous session lifecycle #310).

Run integrity

• Tranche 3 authored blind: first commit of this branch, before any boot or baseline read.
• Candidate config + observability committed before the first gate prompt; zero code
  changes after (evidence commits only).
• One attempt per prompt. The single retry (G9) was pure infra — the host server process
  was externally reaped before the prompt was submitted (noted in the row). F2/F5/G2 were
  re-OPENED (stored apps) only to correct the capture artifact, never re-generated.
• Dedicated headless Chromium (Playwright 1.61.1), one host booted at a time, production
  next start only.

What's in this PR

• docs/eval/runs/2026-07-21/ — README per half (verdict tables + fails-by-class +
  timing + movement vs baseline), 100+ screenshots, aria snapshots, design-pairwise
  verdicts (JSON + table).
• docs/eval/GOLDEN.md — ledger updated with the official numbers; Tranche 3 frozen with
  its 5/10 baseline.
• Gate candidate config (pipeline: { promptRewrite: true, endPass: true }) in both demo
  hosts + the mechanical createVendo apps.pipeline / apps.onPipeline passthroughs
  (the knobs existed on createApps but had no host seam; onPipeline is pure reporting).

pnpm build && pnpm test && pnpm typecheck && pnpm lint all green.

---

---

Summary by cubic

Ships the v4 final gate eval: adds full run evidence and updates the ledger. Eval ran with the candidate pipeline flags ON; the flags are reverted in this branch (not adopted). Official scores — Frozen 30: 17/30; Tranche 2: 7/10; Tranche 3: 5/10; design pairwise: 17W/13L/10T.

• New Features
  • Ran the gate with apps.pipeline: { promptRewrite: true, endPass: true } in apps/demo-bank and apps/demo-accounting; reverted the flags before merge (not adopted).
  • Added onPipeline server-log diagnostics for per-stage reporting (adoption and repair engagement).
  • Updated docs/eval/GOLDEN.md and added run artifacts under docs/eval/runs/2026-07-21/ (screenshots, ARIA snapshots, pairwise results); Tranche 3 frozen at 5/10 baseline.

^{Written for commit ca77b85. Summary will update on new commits.}

[greptile-apps]

Too many files changed for review. (131 files found, 100 file limit)

Bypass the limit by tagging @greptile-apps to review.

[devin-ai-integration]

Devin Review found 1 potential issue.

[devin-ai-integration]

📝 Info: Complete and consistent pipeline/onPipeline passthrough chain

The functional change is a mechanical passthrough of two new host knobs (pipeline, onPipeline) from createVendo config down to the generation engine. I verified the full chain end to end: CreateVendoConfig.apps.pipeline/onPipeline (packages/vendo/src/server.ts:315-318) → forwarded to createApps (packages/vendo/src/server.ts:1274-1275) → AppsConfig.onPipeline and the pre-existing AppsConfig.pipeline (packages/apps/src/runtime.ts:170-174) → spread into GenerationDependencies (packages/apps/src/runtime.ts:554-555) → consumed by the engine/pipeline (packages/apps/src/engine.ts:131-134, packages/apps/src/pipeline.ts:990, pipeline.ts:994). The underlying PipelineConfig type supports both promptRewrite and endPass (packages/apps/src/pipeline.ts:27-35), matching the demo host configs. Both spreads correctly guard on undefined so an unset knob adds no behavior. No gaps or inconsistencies between the two forwarding sites.

---

Was this helpful? React with 👍 or 👎 to provide feedback.