Polish: audit follow-ups (CLAIMS wording, evidence hygiene, cwd-robust scorers)

CLAIMS.md

@@ -29,7 +29,7 @@ same data split, same scorer across systems.
 | Claim | Result | Scorer | Evidence |
 |---|---|---|---|
 | Head-to-head vs. Agentless (Pareto) | Code-Capsules 136/300 resolved @ \$0.436/inst vs. Agentless 125/300 @ \$0.452 — **more resolved at lower cost** | [`benchmarks/swebench/h2h_combine_score.py`](benchmarks/swebench/h2h_combine_score.py) | [`evals/leakfree/tb_forcestage2_first150.jsonl`](evals/leakfree/tb_forcestage2_first150.jsonl), [`tb_forcestage2_second150.jsonl`](evals/leakfree/tb_forcestage2_second150.jsonl), [`evals/h2h_agentless_sonnet_n150_run1_20260528.jsonl`](evals/h2h_agentless_sonnet_n150_run1_20260528.jsonl) |
-| Calibrated cross-tier Pareto menu | cost-min 57 @ \$0.48 · balanced 66 @ \$0.93 · quality 77 @ \$0.91 · quality-max 92 @ \$0.78 · ceiling 96 @ \$0.94 (quality-max out-resolves quality at lower \$/resolve) | [`benchmarks/swebench/cross_tier_pareto_menu.py`](benchmarks/swebench/cross_tier_pareto_menu.py) | [`evals/p10_sonnet_p8c_n150_20260515T235913.jsonl`](evals/p10_sonnet_p8c_n150_20260515T235913.jsonl), [`evals/p9_opus_implicit20_n150_combined_20260527.jsonl`](evals/p9_opus_implicit20_n150_combined_20260527.jsonl) |
+| Calibrated cross-tier Pareto menu | cost-min 57 @ \$0.48 · balanced 66 @ \$0.93 · quality 77 @ \$0.91 · quality-max 92 @ \$0.78 · ceiling 96 @ \$0.94 (quality-max out-resolves quality at lower \$/resolve) | [`benchmarks/swebench/cross_tier_pareto_menu.py`](benchmarks/swebench/cross_tier_pareto_menu.py) | one committed JSONL per cell: cost-min [`evals/p10_sonnet_p8c_n150_20260515T235913.jsonl`](evals/p10_sonnet_p8c_n150_20260515T235913.jsonl), balanced [`evals/leakfree/tb_forcestage2_first150.jsonl`](evals/leakfree/tb_forcestage2_first150.jsonl), quality (leak-free re-run) [`evals/leakfree/exp4_lever_floor100_siginject.jsonl`](evals/leakfree/exp4_lever_floor100_siginject.jsonl), quality-max [`evals/p9_opus_implicit20_n150_combined_20260527.jsonl`](evals/p9_opus_implicit20_n150_combined_20260527.jsonl), ceiling [`evals/p9_opus_floor_n150_combined_20260527.jsonl`](evals/p9_opus_floor_n150_combined_20260527.jsonl) |
 
 ## Mechanism claims
 
@@ -81,8 +81,9 @@ evidence [`evals/cross_vendor_humaneval_n164_postfix_20260525.csv`](evals/cross_
 [`evals/cross_vendor_mbpp_n500_postfix_20260525.csv`](evals/cross_vendor_mbpp_n500_postfix_20260525.csv).
 
 **C12. Plan-then-execute helps every model; critique helps only some.**
-Floor → plan-then-execute is +7pp on Sonnet and +7pp on gpt-5-codex;
-two-pass critique is flat on codex. Scorer
+Over the signaled-budget baseline, plan-then-execute adds +6.7pp on both
+Sonnet (37.3 → 44.0%) and gpt-5-codex (20.7 → 27.3%); two-pass critique
+adds further on Sonnet but is flat on gpt-5-codex (−0.7pp). Scorer
 [`benchmarks/swebench/claim12_plan_critique_lift.py`](benchmarks/swebench/claim12_plan_critique_lift.py).
 
 ## Negative results
@@ -106,9 +107,15 @@ scored 0/77; the canonical `bin/test` harness recovers Agentless to
 31/77. Scorer
 [`benchmarks/swebench/sympy_evalbug_gate.py`](benchmarks/swebench/sympy_evalbug_gate.py).
 
-**C11. Tier escalation has steeply diminishing economics.** Over a
-\$1.40/resolve base, recovering additional resolves by escalating tiers
-costs \$2.20, then \$3.07, then \$7.51 per recovered resolve. Scorer
+**C11. Tier escalation has steeply diminishing economics.** Against a
+\$1.40/resolve base (the 2-config diverse-select), escalating an
+agreement-doomed set to a stronger tier costs far more per recovered
+resolve — and the figure depends on *which* doomed set you escalate, not
+on a single sequential cascade. Escalating the Haiku-doomed set to Opus
+is the cheapest at ~\$2.20/recovered resolve; escalating the Sonnet-doomed
+set to Opus costs ~\$3.07–\$7.51 depending on how many configs you run.
+These are distinct, non-comparable escalation options over different
+doomed sets. Scorer
 [`benchmarks/swebench/value_of_resolve.py`](benchmarks/swebench/value_of_resolve.py).
 
 ## Reproducibility notes

benchmarks/build_evidence_index.py

@@ -26,8 +26,8 @@
 OUT = ROOT / "benchmarks" / "explorer" / "evidence_index.json"
 
 SOURCES = ["evals/*.jsonl", "evals/leakfree/*.jsonl", "evals/*.csv"]
-SKIP = ("bak", "invalid", "contaminated", "raw_with_errors", "credit_exhausted",
-        "disk_corrupted", "preeval_bug", "dropped", ".preeval")
+SKIP = ("bak", "invalid", "contaminated", "ratelimited", "raw_with_errors",
+        "credit_exhausted", "disk_corrupted", "preeval_bug", "dropped", ".preeval")
 
 RESOLVED_KEYS = ("resolved", "shipped_gold_resolved", "gold_resolved")
 COST_KEYS = ("cost_usd", "lever_cost_usd")

benchmarks/explorer/evidence_index.json

@@ -1,6 +1,6 @@
 {
  "note": "Eval-run catalog for the evidence explorer. Generated by benchmarks/build_evidence_index.py from evals/*.jsonl + evals/leakfree/*.jsonl + evals/*.csv; backing claims from claims_results.json.",
- "n_files": 211,
+ "n_files": 210,
  "n_claim_evidence": 29,
  "claims": [
   {
@@ -2828,55 +2828,6 @@
    "claims": [],
    "kind": "supporting"
   },
-  {
-   "path": "evals/leakfree/tb_forcestage2_second150.RATELIMITED-MIXED.jsonl",
-   "label": "force-stage2 two-pass (leak-free) . held-out (second 150)",
-   "format": "jsonl",
-   "n_rows": 150,
-   "resolved": 20,
-   "resolved_pct": 13.3,
-   "resolved_key": "resolved",
-   "mean_cost": 0.1402,
-   "repos": [
-    "psf/requests",
-    "pydata/xarray",
-    "pylint-dev/pylint",
-    "pytest-dev/pytest",
-    "scikit-learn/scikit-learn",
-    "sphinx-doc/sphinx",
-    "sympy/sympy"
-   ],
-   "columns": [
-    "actual_turns",
-    "cost_usd",
-    "duration_s",
-    "effective_input_tokens",
-    "error",
-    "escalated",
-    "escalation_reason",
-    "eval_note",
-    "eval_stdout",
-    "has_patch",
-    "instance_id",
-    "mode",
-    "prompt_budget_hint",
-    "repo",
-    "resolved",
-    "routing_decision",
-    "signals_fired",
-    "stage1_eval_note",
-    "stage1_eval_stdout",
-    "stage1_resolved",
-    "stage1_returncode",
-    "stage1_turns",
-    "stage2_turns",
-    "total_output_tokens",
-    "turn_budget"
-   ],
-   "size_bytes": 155629,
-   "claims": [],
-   "kind": "supporting"
-  },
   {
    "path": "evals/leakfree/warmstart_opus_loc.jsonl",
    "label": "Opus . warm-start",

benchmarks/swebench/agreement_signal.py

@@ -15,12 +15,19 @@
 import glob
 import json
 from itertools import combinations
+from pathlib import Path
+
+# Anchor every eval glob on the repo root so the scorer reproduces from ANY
+# working directory. verify_criteria.py invokes it with cwd=ROOT, but a reader
+# running the bare `python3 benchmarks/swebench/agreement_signal.py` from
+# elsewhere would otherwise silently get empty results.
+ROOT = Path(__file__).resolve().parents[2]
 
 
 def load_configs(pattern="evals/p10_sonnet_*n150*.jsonl"):
     import re
     configs = {}
-    for f in sorted(glob.glob(pattern)):
+    for f in sorted(glob.glob(str(ROOT / pattern))):
         # EXCLUDE same-strategy variance reruns (implicit40_var2/var3, twopass_*_var2/var3):
         # a reroll resolving an instance is sampling noise, not a recoverable strategy, and
         # absorbing it into the reference set spuriously depresses abandon precision (98.5%->90.8%).
@@ -102,7 +109,7 @@ def opus_tier_boundary():
     0/65), so robust to any siginject give-ups."""
     import glob as _g, json as _j
     def L(p):
-        f = sorted(x for x in _g.glob(p) if "bak" not in x and "invalid" not in x)[0]
+        f = sorted(x for x in _g.glob(str(ROOT / p)) if "bak" not in x and "invalid" not in x)[0]
         d = {}
         for line in open(f):
             line = line.strip()
@@ -117,7 +124,7 @@ def L(p):
     # recovery; the 05-27 re-run graded 18 of those 19 (11 resolved), 0 regressions.
     # Globbing 2026052[67]* + sorted()[0] silently picked the incomplete 05-26 file.
     of = L("evals/p9_opus_floor_n150_combined_20260527.jsonl")
-    osig_files = _g.glob("evals/opus_siginject_agreementfail65.jsonl")
+    osig_files = _g.glob(str(ROOT / "evals/opus_siginject_agreementfail65.jsonl"))
     if not osig_files:
         print("\n=== OPUS-TIER boundary: opus siginject sweep not found, skipping ==="); return
     osig = L("evals/opus_siginject_agreementfail65.jsonl")
@@ -146,7 +153,7 @@ def haiku_to_high_escalation():
     collapses on Haiku). Answers 'did we try Haiku->Opus?' (2026-06-04)."""
     import glob as _g, json as _j
     def L(p):
-        fs=[x for x in _g.glob(p) if 'bak' not in x and 'invalid' not in x]
+        fs=[x for x in _g.glob(str(ROOT / p)) if 'bak' not in x and 'invalid' not in x]
         if not fs: return None
         d={}
         for line in open(sorted(fs)[0]):

benchmarks/swebench/value_of_resolve.py

@@ -12,10 +12,16 @@
 
 import glob
 import json
+from pathlib import Path
+
+# Anchor eval globs on the repo root so the scorer reproduces from any CWD
+# (verify_criteria.py runs it with cwd=ROOT; a bare standalone invocation from
+# elsewhere would otherwise silently get empty results).
+ROOT = Path(__file__).resolve().parents[2]
 
 
 def _L(pattern):
-    fs = [x for x in glob.glob(pattern) if "bak" not in x and "invalid" not in x]
+    fs = [x for x in glob.glob(str(ROOT / pattern)) if "bak" not in x and "invalid" not in x]
     if not fs:
         return None
     f = max(fs, key=lambda p: sum(1 for _ in open(p)))

tests/test_runtime_e2e_replay.py

@@ -27,8 +27,9 @@
 from code_capsules.controller.runtime import CodeCapsulesRunner, RunnerPolicy
 
 ROOT = Path(__file__).resolve().parents[1]
-# Exact paths (NOT globs): there is a quarantined *.CONTAMINATED-MIXED.jsonl in
-# the same directory that a glob would sort ahead of the clean held-out file.
+# Exact paths (NOT globs): evals/leakfree/ also holds a quarantined
+# *.RATELIMITED-MIXED.jsonl that a broad *.jsonl glob would pick up alongside
+# the clean held-out files.
 FIRST150 = ROOT / "evals/leakfree/exp4_lever_floor100_siginject.jsonl"
 HELD_OUT = ROOT / "evals/leakfree/exp4_lever_second150.jsonl"