From b9e1f56f4c1cc0b0f41c214b9d75f5a6dde150b7 Mon Sep 17 00:00:00 2001 From: Aninda Ray Date: Sun, 14 Jun 2026 21:14:50 -0700 Subject: [PATCH] Polish: audit follow-ups (CLAIMS wording, evidence hygiene, cwd-robust scorers) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit From the artifact stress-test audit. No reproduced number changed; all data was already correct — these fix wording, evidence-explorer hygiene, and scorer portability. - CLAIMS.md C11: reword escalation economics as three distinct options over different doomed sets (Haiku-doomed ~$2.20; Sonnet-doomed ~$3.07-$7.51), not a single sequential cascade (the paper's Table 11 was already correct). - CLAIMS.md C12: baseline is the signaled-budget cell, not "Floor"; state the precise +6.7pp lifts (Sonnet 37.3->44.0%, codex 20.7->27.3%). - CLAIMS.md menu row: name the source JSONL for every cell, including the leak-free "quality 77" file (exp4_lever_floor100_siginject.jsonl) that was unnamed. - build_evidence_index.py: add "ratelimited" to SKIP so the quarantined RATELIMITED-MIXED held-out file no longer surfaces in the evidence explorer; regenerate evidence_index.json. - test_runtime_e2e_replay.py: fix stale comment that named a nonexistent *.CONTAMINATED-MIXED.jsonl (actual quarantined file is *.RATELIMITED-MIXED.jsonl). - agreement_signal.py, value_of_resolve.py: anchor evals/ globs on the repo root so these standalone-invokable scorers reproduce from any working directory. Verified: 12/12 claims reproduce; both scorers produce correct numbers from a foreign CWD; offline suite 483 passed, 1 skipped, 3 deselected. Co-Authored-By: Claude Opus 4.8 (1M context) --- CLAIMS.md | 19 ++++++--- benchmarks/build_evidence_index.py | 4 +- benchmarks/explorer/evidence_index.json | 51 +------------------------ benchmarks/swebench/agreement_signal.py | 15 ++++++-- benchmarks/swebench/value_of_resolve.py | 8 +++- tests/test_runtime_e2e_replay.py | 5 ++- 6 files changed, 37 insertions(+), 65 deletions(-) diff --git a/CLAIMS.md b/CLAIMS.md index db9e24d..ea71a32 100644 --- a/CLAIMS.md +++ b/CLAIMS.md @@ -29,7 +29,7 @@ same data split, same scorer across systems. | Claim | Result | Scorer | Evidence | |---|---|---|---| | Head-to-head vs. Agentless (Pareto) | Code-Capsules 136/300 resolved @ \$0.436/inst vs. Agentless 125/300 @ \$0.452 — **more resolved at lower cost** | [`benchmarks/swebench/h2h_combine_score.py`](benchmarks/swebench/h2h_combine_score.py) | [`evals/leakfree/tb_forcestage2_first150.jsonl`](evals/leakfree/tb_forcestage2_first150.jsonl), [`tb_forcestage2_second150.jsonl`](evals/leakfree/tb_forcestage2_second150.jsonl), [`evals/h2h_agentless_sonnet_n150_run1_20260528.jsonl`](evals/h2h_agentless_sonnet_n150_run1_20260528.jsonl) | -| Calibrated cross-tier Pareto menu | cost-min 57 @ \$0.48 · balanced 66 @ \$0.93 · quality 77 @ \$0.91 · quality-max 92 @ \$0.78 · ceiling 96 @ \$0.94 (quality-max out-resolves quality at lower \$/resolve) | [`benchmarks/swebench/cross_tier_pareto_menu.py`](benchmarks/swebench/cross_tier_pareto_menu.py) | [`evals/p10_sonnet_p8c_n150_20260515T235913.jsonl`](evals/p10_sonnet_p8c_n150_20260515T235913.jsonl), [`evals/p9_opus_implicit20_n150_combined_20260527.jsonl`](evals/p9_opus_implicit20_n150_combined_20260527.jsonl) | +| Calibrated cross-tier Pareto menu | cost-min 57 @ \$0.48 · balanced 66 @ \$0.93 · quality 77 @ \$0.91 · quality-max 92 @ \$0.78 · ceiling 96 @ \$0.94 (quality-max out-resolves quality at lower \$/resolve) | [`benchmarks/swebench/cross_tier_pareto_menu.py`](benchmarks/swebench/cross_tier_pareto_menu.py) | one committed JSONL per cell: cost-min [`evals/p10_sonnet_p8c_n150_20260515T235913.jsonl`](evals/p10_sonnet_p8c_n150_20260515T235913.jsonl), balanced [`evals/leakfree/tb_forcestage2_first150.jsonl`](evals/leakfree/tb_forcestage2_first150.jsonl), quality (leak-free re-run) [`evals/leakfree/exp4_lever_floor100_siginject.jsonl`](evals/leakfree/exp4_lever_floor100_siginject.jsonl), quality-max [`evals/p9_opus_implicit20_n150_combined_20260527.jsonl`](evals/p9_opus_implicit20_n150_combined_20260527.jsonl), ceiling [`evals/p9_opus_floor_n150_combined_20260527.jsonl`](evals/p9_opus_floor_n150_combined_20260527.jsonl) | ## Mechanism claims @@ -81,8 +81,9 @@ evidence [`evals/cross_vendor_humaneval_n164_postfix_20260525.csv`](evals/cross_ [`evals/cross_vendor_mbpp_n500_postfix_20260525.csv`](evals/cross_vendor_mbpp_n500_postfix_20260525.csv). **C12. Plan-then-execute helps every model; critique helps only some.** -Floor → plan-then-execute is +7pp on Sonnet and +7pp on gpt-5-codex; -two-pass critique is flat on codex. Scorer +Over the signaled-budget baseline, plan-then-execute adds +6.7pp on both +Sonnet (37.3 → 44.0%) and gpt-5-codex (20.7 → 27.3%); two-pass critique +adds further on Sonnet but is flat on gpt-5-codex (−0.7pp). Scorer [`benchmarks/swebench/claim12_plan_critique_lift.py`](benchmarks/swebench/claim12_plan_critique_lift.py). ## Negative results @@ -106,9 +107,15 @@ scored 0/77; the canonical `bin/test` harness recovers Agentless to 31/77. Scorer [`benchmarks/swebench/sympy_evalbug_gate.py`](benchmarks/swebench/sympy_evalbug_gate.py). -**C11. Tier escalation has steeply diminishing economics.** Over a -\$1.40/resolve base, recovering additional resolves by escalating tiers -costs \$2.20, then \$3.07, then \$7.51 per recovered resolve. Scorer +**C11. Tier escalation has steeply diminishing economics.** Against a +\$1.40/resolve base (the 2-config diverse-select), escalating an +agreement-doomed set to a stronger tier costs far more per recovered +resolve — and the figure depends on *which* doomed set you escalate, not +on a single sequential cascade. Escalating the Haiku-doomed set to Opus +is the cheapest at ~\$2.20/recovered resolve; escalating the Sonnet-doomed +set to Opus costs ~\$3.07–\$7.51 depending on how many configs you run. +These are distinct, non-comparable escalation options over different +doomed sets. Scorer [`benchmarks/swebench/value_of_resolve.py`](benchmarks/swebench/value_of_resolve.py). ## Reproducibility notes diff --git a/benchmarks/build_evidence_index.py b/benchmarks/build_evidence_index.py index 01360b0..2006b7b 100644 --- a/benchmarks/build_evidence_index.py +++ b/benchmarks/build_evidence_index.py @@ -26,8 +26,8 @@ OUT = ROOT / "benchmarks" / "explorer" / "evidence_index.json" SOURCES = ["evals/*.jsonl", "evals/leakfree/*.jsonl", "evals/*.csv"] -SKIP = ("bak", "invalid", "contaminated", "raw_with_errors", "credit_exhausted", - "disk_corrupted", "preeval_bug", "dropped", ".preeval") +SKIP = ("bak", "invalid", "contaminated", "ratelimited", "raw_with_errors", + "credit_exhausted", "disk_corrupted", "preeval_bug", "dropped", ".preeval") RESOLVED_KEYS = ("resolved", "shipped_gold_resolved", "gold_resolved") COST_KEYS = ("cost_usd", "lever_cost_usd") diff --git a/benchmarks/explorer/evidence_index.json b/benchmarks/explorer/evidence_index.json index 1c683b6..fecb5e8 100644 --- a/benchmarks/explorer/evidence_index.json +++ b/benchmarks/explorer/evidence_index.json @@ -1,6 +1,6 @@ { "note": "Eval-run catalog for the evidence explorer. Generated by benchmarks/build_evidence_index.py from evals/*.jsonl + evals/leakfree/*.jsonl + evals/*.csv; backing claims from claims_results.json.", - "n_files": 211, + "n_files": 210, "n_claim_evidence": 29, "claims": [ { @@ -2828,55 +2828,6 @@ "claims": [], "kind": "supporting" }, - { - "path": "evals/leakfree/tb_forcestage2_second150.RATELIMITED-MIXED.jsonl", - "label": "force-stage2 two-pass (leak-free) . held-out (second 150)", - "format": "jsonl", - "n_rows": 150, - "resolved": 20, - "resolved_pct": 13.3, - "resolved_key": "resolved", - "mean_cost": 0.1402, - "repos": [ - "psf/requests", - "pydata/xarray", - "pylint-dev/pylint", - "pytest-dev/pytest", - "scikit-learn/scikit-learn", - "sphinx-doc/sphinx", - "sympy/sympy" - ], - "columns": [ - "actual_turns", - "cost_usd", - "duration_s", - "effective_input_tokens", - "error", - "escalated", - "escalation_reason", - "eval_note", - "eval_stdout", - "has_patch", - "instance_id", - "mode", - "prompt_budget_hint", - "repo", - "resolved", - "routing_decision", - "signals_fired", - "stage1_eval_note", - "stage1_eval_stdout", - "stage1_resolved", - "stage1_returncode", - "stage1_turns", - "stage2_turns", - "total_output_tokens", - "turn_budget" - ], - "size_bytes": 155629, - "claims": [], - "kind": "supporting" - }, { "path": "evals/leakfree/warmstart_opus_loc.jsonl", "label": "Opus . warm-start", diff --git a/benchmarks/swebench/agreement_signal.py b/benchmarks/swebench/agreement_signal.py index 36d0a2d..edda228 100644 --- a/benchmarks/swebench/agreement_signal.py +++ b/benchmarks/swebench/agreement_signal.py @@ -15,12 +15,19 @@ import glob import json from itertools import combinations +from pathlib import Path + +# Anchor every eval glob on the repo root so the scorer reproduces from ANY +# working directory. verify_criteria.py invokes it with cwd=ROOT, but a reader +# running the bare `python3 benchmarks/swebench/agreement_signal.py` from +# elsewhere would otherwise silently get empty results. +ROOT = Path(__file__).resolve().parents[2] def load_configs(pattern="evals/p10_sonnet_*n150*.jsonl"): import re configs = {} - for f in sorted(glob.glob(pattern)): + for f in sorted(glob.glob(str(ROOT / pattern))): # EXCLUDE same-strategy variance reruns (implicit40_var2/var3, twopass_*_var2/var3): # a reroll resolving an instance is sampling noise, not a recoverable strategy, and # absorbing it into the reference set spuriously depresses abandon precision (98.5%->90.8%). @@ -102,7 +109,7 @@ def opus_tier_boundary(): 0/65), so robust to any siginject give-ups.""" import glob as _g, json as _j def L(p): - f = sorted(x for x in _g.glob(p) if "bak" not in x and "invalid" not in x)[0] + f = sorted(x for x in _g.glob(str(ROOT / p)) if "bak" not in x and "invalid" not in x)[0] d = {} for line in open(f): line = line.strip() @@ -117,7 +124,7 @@ def L(p): # recovery; the 05-27 re-run graded 18 of those 19 (11 resolved), 0 regressions. # Globbing 2026052[67]* + sorted()[0] silently picked the incomplete 05-26 file. of = L("evals/p9_opus_floor_n150_combined_20260527.jsonl") - osig_files = _g.glob("evals/opus_siginject_agreementfail65.jsonl") + osig_files = _g.glob(str(ROOT / "evals/opus_siginject_agreementfail65.jsonl")) if not osig_files: print("\n=== OPUS-TIER boundary: opus siginject sweep not found, skipping ==="); return osig = L("evals/opus_siginject_agreementfail65.jsonl") @@ -146,7 +153,7 @@ def haiku_to_high_escalation(): collapses on Haiku). Answers 'did we try Haiku->Opus?' (2026-06-04).""" import glob as _g, json as _j def L(p): - fs=[x for x in _g.glob(p) if 'bak' not in x and 'invalid' not in x] + fs=[x for x in _g.glob(str(ROOT / p)) if 'bak' not in x and 'invalid' not in x] if not fs: return None d={} for line in open(sorted(fs)[0]): diff --git a/benchmarks/swebench/value_of_resolve.py b/benchmarks/swebench/value_of_resolve.py index 1d8d22c..a9eb6f8 100644 --- a/benchmarks/swebench/value_of_resolve.py +++ b/benchmarks/swebench/value_of_resolve.py @@ -12,10 +12,16 @@ import glob import json +from pathlib import Path + +# Anchor eval globs on the repo root so the scorer reproduces from any CWD +# (verify_criteria.py runs it with cwd=ROOT; a bare standalone invocation from +# elsewhere would otherwise silently get empty results). +ROOT = Path(__file__).resolve().parents[2] def _L(pattern): - fs = [x for x in glob.glob(pattern) if "bak" not in x and "invalid" not in x] + fs = [x for x in glob.glob(str(ROOT / pattern)) if "bak" not in x and "invalid" not in x] if not fs: return None f = max(fs, key=lambda p: sum(1 for _ in open(p))) diff --git a/tests/test_runtime_e2e_replay.py b/tests/test_runtime_e2e_replay.py index 1532b3c..407eb20 100644 --- a/tests/test_runtime_e2e_replay.py +++ b/tests/test_runtime_e2e_replay.py @@ -27,8 +27,9 @@ from code_capsules.controller.runtime import CodeCapsulesRunner, RunnerPolicy ROOT = Path(__file__).resolve().parents[1] -# Exact paths (NOT globs): there is a quarantined *.CONTAMINATED-MIXED.jsonl in -# the same directory that a glob would sort ahead of the clean held-out file. +# Exact paths (NOT globs): evals/leakfree/ also holds a quarantined +# *.RATELIMITED-MIXED.jsonl that a broad *.jsonl glob would pick up alongside +# the clean held-out files. FIRST150 = ROOT / "evals/leakfree/exp4_lever_floor100_siginject.jsonl" HELD_OUT = ROOT / "evals/leakfree/exp4_lever_second150.jsonl"