feat(trace-repair): measure what unconditional continuation rescues by drewstone · Pull Request #558 · tangle-network/agent-eval

drewstone · 2026-08-10T09:46:18Z

Admission condition 3 asks whether a row is rescued by continuing from the recorded end state with no intervention.
Both milestone runs answered it under a control pinned to zero model calls, so the container the control graded held the same bytes the end-state check had already graded as failing.
On the 32 rows this branch measures, that control returned 0 passes out of 96 rollouts — the only answer it could return.

This adds the runner that asks the question with a budget: the pinned mini-swe-agent policy, a real model, no analyst, no hint, no gate, no injected action.

Depends on #556 (branched from fix/admission-calibration), which supplies the declared control policy and the oracle-determinism certification this run reads.

What runs

Three container generations per rollout, because three constraints point in different directions:

generation	network	why
replay	default bridge	the recording had internet, and the milestone replays used it
continue	`none`	the pinned policy requires it, and `describe()` re-reads the mode from the daemon
grade	default bridge	every task's `test.sh` runs `apt-get`, `curl` and `uvx` before its first assertion

State moves between generations as a committed image, because Docker refuses to attach a network to a container created in none mode:

Error response from daemon: container cannot be connected to multiple networks
with one of the networks in private (none) mode

Isolation is proven rather than assumed — inside the continuation container:

DNS FAIL
bash: connect: Network is unreachable
TCP_DOWN

and in a networked control container from the same image: DNS_OK, TCP_UP.

Design points that are load-bearing

The stop point is built once per row and shared by that row's rollouts, so the only thing varying within a row is the continuation. It also pays the expensive commit once: a recording that ran pip install datasets transformers writes a 785 MB layer that takes minutes to commit.
Stop points survive between passes. A campaign that adds rollouts pass by pass must hand every pass the same stop point, or its passes differ by their replay as well as by their rollout. The image carries the container state; a record beside it carries the message list and divergence count the image cannot.
The slug that addresses a stop point is checked for collisions across the selected rows before anything is built, because two rows sharing one would share a container state silently.
Stop points build without the measurement seat, since reconstructing them needs containers and no model call.
Cost is attributed per call, not from the account counter. GET /v1/credits is account-wide; while the pilot ran, 18 other processes on this host were calling the same key (12 deepseek-v4-pro, 6 glm-5.2). A rollout with one unpriced call reports uncaptured, never a partial sum.

Replay fidelity, measured

32 stop points, 0 errors, 3 divergences across 128 replayed steps (2.34 %).
The two rows carrying them reproduce milestone 2's recorded ratios exactly — password-recovery__3ppLoVS at 2/20 = 0.100 and password-recovery__rGg4KKB at 1/11 = 0.091 — which is an independent check that this replay lands where the milestone's did.

Pre-registration

Design, denominator chain, budget rule and threats were written to PREREG.md before the first rollout, and every later change is an appended amendment carrying its timestamp and its reason. Two amendments were made on cost evidence with no outcome in existence, verified by test -f on the report path at the time of writing.

Task exclusion is read from the certification #556 adds: largest-eigenval is excluded as NONDETERMINISTIC_ORACLE (flip rate 37.5 %, 16 replicates), so 5 of the 48 sampled rows never run.

Proof

tsc --noEmit -p tsconfig.script.json     clean
pnpm vitest run tests/trace-repair src/trace-repair
  Test Files  16 passed (16)      Tests  239 passed (239)
git merge-tree --write-tree origin/main HEAD   merges cleanly

The measurement itself runs behind the shared measurement seat and reports separately.

… a task whose grader is a coin flip Admission condition 3 asks whether a row is rescued by continuing from the recorded end state with no intervention. Both milestone runs answered it under a control pinned to zero model calls. A rollout that makes no model call executes no command, so both control arms graded the bytes condition 2 had already graded as failing: the condition could only ever fire on a grader that disagreed with itself, and every row walked through it. Nothing in the artifacts recorded which control had screened a row. The control is now a hashed declaration rather than a default. `defineControlPolicy` carries the step budget, scaffold, model and command timeout into a digest, and `assertControlCalibrated` refuses the two contradictions at configuration time: screening under a control that cannot act, and calling a control inert when it can. Under `controlScreening: 'declared-inert'` a control pass is recorded as `control-passed-on-identical-state` rather than as a rescue nothing performed. Every admission decision, admitted or not, carries the policy, its digest, the screening mode and the task's measured oracle flip rate. A task graded by wall clock cannot serve as ground truth for an intervention study. `certify-task-oracle.sh` gains a determinism phase: re-grade byte-identical containers N times, idle and under CPU contention, and count flips per assertion rather than per suite. Counting the suite reward hides the failure, because the reward is a conjunction and a suite whose per-parameter timing assertions each flip still returns one steady verdict at a state far from the threshold. The verdict rule lives in `oracleDeterminism` and is applied through `scripts/tb-oracle-determinism.ts`, so a certification run and a campaign cannot disagree about what a stable oracle is. Skipping the phase yields CERTIFIED_UNCHECKED_DETERMINISM, never CERTIFIED. Measured at the image digests the milestones ran against, 16 replicates per task: password-recovery, sanitize-git-repo and count-dataset-tokens flip 0 units; largest-eigenval flips 8 of its 9 test_speedup[size] parameters, worst 37.5 %, while its whole-suite reward never moves. The measurements are checked in as benchmarks/trace-repair/task-oracles.json, which stores replicates rather than verdicts so the file cannot declare a task stable. Admission reads that certification before it opens a container. A task with no entry leaves as task-oracle-uncertified; an unstable one as task-oracle-nondeterministic. Rows screened by an inert control carry the control-cannot-rescue threat into the Delta-repair report. Milestone 1's headline is unchanged and its published numbers are not rewritten; docs/trace-repair-admission.md states which of its rows and verdicts the defects touched.

…w a flip A load group of one measures nothing, and the substrate rule already rejects it. Refusing at the flag names the mistake instead of failing three phases in.

…arm B alone Admission condition 3 asks whether a row is rescued by continuing from the recorded end state with no intervention. Both milestone runs answered it under a control pinned to zero model calls, which executes no command and therefore grades the bytes the end-state check already graded as failing. This runs that control with a budget: the pinned mini-swe-agent policy, a real model, no analyst, no hint and no gate, over the rows a deterministic oracle can grade. Three container generations per rollout, because the replay needs the network the recording had, the pinned policy requires none, and every task suite runs apt-get and uvx before its first assertion. A container created in none mode cannot be attached to a network, so state moves between generations as a committed image. The stop point is reconstructed once per row and shared by its rollouts, so the only thing varying within a row is the continuation.

…g every model call A campaign that adds rollouts pass by pass runs one invocation per pass and shifts the rollout index, so a later pass draws new seeds rather than redrawing the first one. Each model call writes one line naming the served id, its latency and its token counts, so a run measured in hours shows where it is instead of going dark between rollouts.

… passes A campaign that adds rollouts pass by pass must hand every pass the same stop point, or its passes differ by their replay as well as by their rollout. The image holds the container state and a record beside it holds the message list and divergence count the replay produced, which the image cannot carry. The slug that addresses a stop point is checked for collisions across the selected rows before anything is built, because two rows sharing one would share a container state without saying so.

Reconstructing a recorded stop point needs containers and no model call. A campaign queued behind the seat builds them first, so the seat's time is spent on model calls alone.

tangletools

✅ Auto-approved drewstone PR — `33bf3f91`

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

_{tangletools · auto-approval · reason: drewstone_author · 2026-08-10T09:46:26Z}

…model The router answered a request for deepseek/deepseek-v3.2 with deepseek-v4-flash. A pinned policy that recorded those rollouts would name the wrong model in every number it produced, and pricing by the requested id would bill v4-flash traffic at v3.2 rates. servedMatches compares the final path segment, so a vendor prefix still matches and a different model does not. Cost is priced on the served id.

tangletools

🟡 Value Audit — sound-with-nits


Verdict	sound-with-nits
Concerns	3 (1 low, 2 weak-concern)
Heuristic	0.1s
Duplication	0.1s
Interrogation	504.9s (2 bridge agents)
Total	505.1s

💰 Value — sound

Declares and enforces the control policy an admission row is screened under, and certifies task graders for determinism — fixing a real measurement-integrity flaw where zero-call controls graded the same failing bytes twice; built coherently in the codebase's substrate+script grain.

What it does: Two new substrate modules and one runner. (1) control-policy.ts makes the control a declared, hashed parameter with a calibration rule (assertControlCalibrated) that refuses a screening control which cannot act (stepBudget 0) and refuses an inert label on a control that can. (2) oracle-determinism.ts is a pure rule that re-grades byte-identical containers N times and reads the per-assertion minori
Goals it achieves: (1) Stop every admission row from walking through a control that cannot fire: a zero-call control grades the same bytes the end-state check already failed, so its verdict is the task grader answering twice, not a measurement of rescue. (2) Catch task graders that are not a function of container state (largest-eigenval's wall-clock assertion flips at 37.5%) before they contaminate a campaign. (3) P
Assessment: Sound and in-grain. The new modules are pure substrate (no containers, no models, inject nothing) and sit beside the existing trace-repair primitives. The key integrity property is structural: both admission paths — the pure contract (admission-contract.ts:198) and the executing pre-pass (admission.ts:306) — call the same assertControlCalibrated against the same ControlCapability shape, so they ca
Better / existing approach: none — this is the right approach. Checked: the freelunch-model.ts script uses raw fetch rather than the codebase's ChatClient (src/analyst/chat-client.ts), but ContinuationModel is a deliberately simple (req)=>Promise injection point at continuation-policy.ts:167 and the script is the intended adapter; ChatClient lives in the analyst layer with model-selection and signal machinery the conti
Model: opencode/zai-coding-plan/glm-5.2
Bridge attempts: 2
Bridge warning: opencode/kimi-for-coding/k2p7: opencode: opencode error

🎯 Usefulness — sound-with-nits

A coherent, well-integrated measurement runner that finally asks admission condition 3 with a real model budget instead of a zero-step control; built in the exact grain of the existing milestone runner, with one minor typecheck-coverage gap in the new scripts.

Integration: The two new substrate modules (control-policy.ts, oracle-determinism.ts) are exported from src/trace-repair/index.ts:168-178 and :243-262 and consumed everywhere they should be: admitRow (admission-contract.ts:198-216 asserts control calibration and reads oracle certification), runAdmission (admission.ts:294-310 now requires taskOracles and calls assertControlCalibrated), grade.ts:93-95 carries co
Fit with existing patterns: No competing pattern exists. The freelunch runner mirrors milestone1.ts structurally: same Docker session helper, same loadSuite/taskOracle, same stepTimeoutMs/recordedReturncode helpers, same RolloutOutcome shape, same mapLimit concurrency primitive. The one novel element — three container generations per rollout (replay on default network, continue on --network none, grade back on default networ
Real-world viability: The runner is built for multi-hour real runs: configurable concurrency (mapLimit), per-rollout try/catch that records failures as outcomes rather than aborting, stop-point build failures recorded for every rollout of the row, container cleanup in finally blocks, stop-point reuse across campaign passes via image-inspect + a digest-checked record file (freelunch.ts:378-414), slug-collision detection
Model: opencode/zai-coding-plan/glm-5.2
Bridge attempts: 1

🔎 Heuristic Signals

🟡 Cruft: magic number added scripts/tb-repair-freelunch.ts

 const seconds = Math.ceil(timeoutMs / 1000)

🎯 Usefulness Audit

🟡 New campaign scripts are not in tsconfig.script.json, so CI does not typecheck them [integration] ``

tsconfig.script.json:4 includes only scripts/tb-repair-milestone1.ts in its include array. The three new scripts added by this PR — scripts/tb-repair-freelunch.ts, scripts/tb-repair-freelunch-model.ts, scripts/tb-oracle-determinism.ts — are absent, so pnpm typecheck:scripts skips them entirely. (tsconfig.json includes only src/, so pnpm typecheck does not cover scripts either.) Their imports all resolve against the tested public API and the substrate modules they call are fully tested, so th

💰 Value Audit

🟡 Script-level helpers duplicated between tb-repair-milestone1.ts and tb-repair-freelunch.ts [duplication] ``

stepTimeoutMs, recordedReturncode, loadSuite, loadTask, loadTaskOracles, and docker-session helpers are copy-pasted across the two runners (freelunch.ts:126-261 vs milestone1.ts:104-373). Both scripts/ runners are self-contained operational one-offs and the directory has no shared-helpers module today, so this is expected for scripts/ — but a third tb-repair runner would make extraction worthwhile. Not gating; note for when a third arrives.

What this audit checks

It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.

Pass	What it asks
Heuristic	Vague title? Whitespace-only or cruft-bearing diff? (content signals only)
Duplication	Do added function/class names already exist elsewhere in the repo?
Value Audit	What does it do? What goal does it achieve? Is it good? Better architecture or already-exists?
Usefulness Audit	Does it integrate and fit? Will it hold up in real use and actually get used?

Findings are concerns, not blocks — the human reviewer decides what to do with them.

_{value-audit · 20260810T102847Z}

3 of 64 rollouts and 2 of 32 rows, against the 0 of 96 a zero-step control was structurally obliged to report. One row was rescued in both its rollouts on a suite that returned the same verdict in all 16 certification replicates, so the rate is not the grader disagreeing with itself. Every rescue came from a rollout that decided it was finished; no rollout that burned all 20 steps ever passed.

drewstone · 2026-08-10T10:31:35Z

Measured: unconditional continuation rescues 3 of 64 rollouts (4.7 %), 2 of 32 rows (6.2 %)

Uniform n = 2 over all 32 rows, glm-5.2 pinned and verified served as itself, 0 harness errors.
On these same rows the zero-step control that screened both milestones returned 0 of 96 — the only answer it could return.

interval, 95 %	rescue rate
task-clustered bootstrap (3 clusters, 10 000 resamples, seed 7)	0.0 % – 10.0 %
row-clustered bootstrap (32 clusters)	0.0 % – 12.5 %
exact Clopper-Pearson on rollouts	1.0 % – 13.1 %

Not noise. count-dataset-tokens__HL3ZzrX was rescued in both its rollouts, on a suite that returned the same verdict in all 16 certification replicates. The recorded agent had the right answer in a form the grader rejected; the continuation found the formatting defect, stripped it, and submitted. A second row solved its task outright — searched a disk image, carved an embedded archive, recovered the password.

task	rows	rollouts	passes	rate	recorded steps using the network
`count-dataset-tokens`	10	20	2	10.0 %	24.3 %
`password-recovery`	6	12	1	8.3 %	0 %
`sanitize-git-repo`	16	32	0	0 %	0 %

exit status	rollouts	passes	rate
`submitted`	15	3	20.0 %
`step-budget-exhausted`	46	0	0 %
`model-error` (provider 503 after 4 retries)	3	0	0 %

Every rescue came from a rollout that decided it was finished; no rollout that burned all 20 steps ever passed.

Cost, priced per call at the router's published rate: $9.4806 for 64 rollouts — $0.1481 per rollout, $0.2963 per row at n = 2, which is what a paired study should budget-match against. Median prompt 170 939 tokens, median completion 6 663, median 20 steps.

Verdict. A high rate would have killed the gated-stop thesis. It did not: unconditional continuation captures a small, real slice of the headroom at a real price. That licenses the paired study; it does not prove a gate would claim the rest. Condition 3 now has a calibration number instead of a structural zero.

Two changes this run forced into the code, both from things that actually happened:

Model substitution. The router answered deepseek/deepseek-v3.2 with deepseek-v4-flash for 29 calls. servedMatches now ends a rollout on a mismatch, and cost is priced on the served id rather than the requested one.
Raw records are credential-bearing. Push protection blocked an earlier commit of the run record: a container carried a Hugging Face user access token in its cached credentials. Records stay out of the repo.

Full report with per-row and per-rollout tables, threats, and the reproduce commands: docs/trace-repair-free-lunch.md.
The pre-registration and its six timestamped amendments are in ~/bench-cache/freelunch-20260810/PREREG.md.

tangletools · 2026-08-10T10:57:07Z

⚠️ Review Interrupted — `33bf3f91`

The review runner stopped before publishing a final verdict: webhook_restarted.

State	Detail
Interrupted	webhook restarted

No review verdict was produced for this run. Trigger a fresh review on the current PR head if the PR is still open.

_{tangletools · #558 · model: kimi-for-coding · updated 2026-08-10T10:57:05Z}

tangletools

✅ Auto-approved drewstone PR — `f1f03f2d`

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

_{tangletools · auto-approval · reason: drewstone_author · 2026-08-10T11:07:12Z}

tangletools

🟢 Value Audit — sound


Verdict	sound
Concerns	2 (1 low, 1 weak-concern)
Heuristic	0.1s
Duplication	0.0s
Interrogation	189.0s (2 bridge agents)
Total	189.1s

💰 Value — sound

Adds the measurement runner and substrate declarations to ask admission condition 3 (does unconditional continuation rescue failed rows?) with a real model budget, exposing that the prior milestone runs answered it under a structurally tautological zero-call control — a high-integrity fix built clea

What it does: Two things. (1) Substrate: adds control-policy.ts (declares what an admission control can do — a hashed ControlPolicy with stepBudget and canRescue — plus assertControlCalibrated, which THROWS on a control/screening-mode contradiction, so a zero-step-budget control paired with enforced screening can no longer pass silently), and oracle-determinism.ts (reduces measured grader replicat
Goals it achieves: Make admission condition 3 actually screenable. Both milestone runs screened rows under a control pinned to zero model calls (zeroStepContinuation in tb-repair-milestone1.ts:131), which executes no command and therefore grades the same bytes the end-state check already graded as failing — structurally obliged to return 0/96 passes. This PR (a) makes that pairing a hard fault via `assertControl
Assessment: Good change, built in the grain of the codebase. The layering is exactly what CLAUDE.md prescribes for this substrate package: pure modules in src/trace-repair/ own what counts and are hashed/auditable; scripts in scripts/ own execution (Docker, models, file I/O). The ControlCapability adapter (control-policy.ts:63-68) lets both the pure contract path (admitRow in `admission-contract.ts:
Better / existing approach: No materially better architecture found. The change extends the existing admission/continuation layer rather than forking it: ControlPolicy projects into ControlCapability, which the contract path already accepts, so the new gate reuses the existing decision machinery. The runner reuses the substrate's runContinuation, definePinnedContinuationPolicy, createDockerContinuationEnvironment,
Model: opencode/zai-coding-plan/glm-5.2
Bridge attempts: 2
Bridge warning: opencode/kimi-for-coding/k2p7: opencode: opencode error

🎯 Usefulness — sound

A coherent, fully-wired measurement that closes a structural gap in admission condition 3, extending the substrate with two modules already consumed by the existing milestone runner and adding a reproducible campaign runner.

Integration: Everything is reachable now. control-policy.ts and oracle-determinism.ts are exported from src/trace-repair/index.ts and consumed by admission-contract.ts:198, admission.ts:306, admission-records.ts, grade.ts:37, and delta-repair.ts, plus the updated tb-repair-milestone1.ts and scripts/tb-oracle-determinism.ts. task-oracles.json is read by parseTaskOracleRegistry in both campaign runners. The new
Fit with existing patterns: Follows the established pattern exactly. scripts/tb-repair-*.ts campaign runners are the convention (tb-repair-milestone1.ts is the peer); the new modules sit in src/trace-repair/ and are consumed upward, matching the CLAUDE.md substrate-layering rule that agent-eval owns the primitives. The machine-specific bench-cache paths and the docker/suite helpers mirror milestone1, so this is the grain, no
Real-world viability: Built for failure paths, not just the happy path. Per-row and per-rollout errors are caught and recorded as outcomes rather than aborting the run (tb-repair-freelunch.ts:701-746); the model layer retries on provider 5xx/429 and records model-error rollouts instead of dropping them, so the rescue rate is an honest lower bound; served-model substitution is detected and stops the rollout (4a51f08); u
Model: opencode/zai-coding-plan/glm-5.2
Bridge attempts: 1

🔎 Heuristic Signals

🟡 Cruft: magic number added scripts/tb-repair-freelunch.ts

 const seconds = Math.ceil(timeoutMs / 1000)

💰 Value Audit

🟡 Campaign scripts re-implement substrate helpers for recorded-returncode parsing and timeout detection [duplication] ``

Both tb-repair-freelunch.ts and tb-repair-milestone1.ts define local recordedReturncode (regex /<returncode>(-?\d+)<\/returncode>/, freelunch:132, milestone1:373) and RECORDED_TIMEOUT_MARKER/stepTimeoutMs (freelunch:83-130, milestone1:102-108). The substrate already exports parseRecordedReturncode (src/trajectory-replay/steps.ts:32) and isRecordedTimeout (src/trace-repair/mini-swe-scaffold.ts:201, re-exported from index.ts:230), and the substrate's own grade.ts uses those

What this audit checks

It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.

Pass	What it asks
Heuristic	Vague title? Whitespace-only or cruft-bearing diff? (content signals only)
Duplication	Do added function/class names already exist elsewhere in the repo?
Value Audit	What does it do? What goal does it achieve? Is it good? Better architecture or already-exists?
Usefulness Audit	Does it integrate and fit? Will it hold up in real use and actually get used?

Findings are concerns, not blocks — the human reviewer decides what to do with them.

_{value-audit · 20260810T111040Z}

tangletools · 2026-08-10T11:39:40Z

⚠️ Review Incomplete — `f1f03f2d`

At least one required reviewer lane failed closed. No approval or request-changes review was published. This is a reviewer run failure, not a PR quality score.

Trigger a fresh review on the current PR head.

_{tangletools · 2026-08-10T11:39:38Z}

tangletools

✅ Auto-approved drewstone PR — `f1f03f2d`

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

_{tangletools · auto-approval · reason: drewstone_author · 2026-08-10T11:47:14Z}

tangletools

🟡 Value Audit — sound-with-nits


Verdict	sound-with-nits
Concerns	3 (1 low, 2 weak-concern)
Heuristic	0.1s
Duplication	0.0s
Interrogation	196.3s (2 bridge agents)
Total	196.4s

💰 Value — sound-with-nits

Fixes a structural flaw where the admission control could never fire (zero-step budget graded identical failing bytes), by declaring/calibrating the control policy and running the measurement with a real model; sound architecture in the codebase's grain, with minor script-level helper duplication.

What it does: Adds two new pure substrate modules to src/trace-repair/. (1) oracle-determinism.ts decides whether a task's own test suite is a deterministic function of container state — it reads measured replicates (idle + contended loads, unsolved + solved states) and computes per-assertion flip rates, refusing to treat a reward-level agreement as stability when individual assertions flip. (2) control-policy.
Goals it achieves: The core goal is measurement integrity: admission condition 3 ('is this row rescued by continuing from the end state with no intervention?') was previously answered under a control pinned to zero model calls, which executes no command and therefore grades the same failing bytes the end-state check already graded — it returned 0/96 on 32 rows because it was structurally incapable of returning anyth
Assessment: Sound and in the grain of the codebase. The two new substrate modules follow the exact pattern the existing trace-repair module established: pure contract (oracle-determinism.ts, control-policy.ts) separated from execution (admission.ts), branded phantom types for type-level ordering invariants (AdmittedRow), fail-loud on configuration faults (assertControlCalibrated throws UncalibratedControlErro
Better / existing approach: Searched for existing Docker campaign infrastructure the runner could reuse: found createDockerContinuationEnvironment and nodeProcessRunner in src/trace-repair/docker-environment.ts, which the runner already imports and uses correctly. The runner's custom dockerSession() helper (for replay and grade containers that need default networking) cannot use the substrate's --network none factory, so it
Model: opencode/zai-coding-plan/glm-5.2
Bridge attempts: 2
Bridge warning: opencode/kimi-for-coding/k2p7: opencode: opencode error

🎯 Usefulness — sound

Adds the real-budget continuation runner admission condition 3 always needed, wired into the core admission contract and following the repo's established campaign-script pattern; it already ran and produced a decision-relevant 4.7% rescue rate.

Integration: Fully reachable, on two paths. (1) Substrate: control-policy.ts and oracle-determinism.ts are imported and REQUIRED by admission-contract.ts:40-44,97-99 — admitRow() refuses evidence without an oracleDeterminism verdict and a declared/hashed controlPolicy, and assertControlCalibrated() gates every decision (admission-contract.ts:198). Both are exported from src/trace-repair/index.ts:168-262. (2) E
Fit with existing patterns: Matches the established grain exactly. Standalone campaign scripts under scripts/ (freelunch mirrors milestone1), pure substrate logic under src/trace-repair/, all re-exported from index.ts. Per CLAUDE.md's layering rule, ControlPolicy and OracleDeterminismVerdict are substrate primitives (they make sense without a running agent loop), so they belong here, not in a consumer. No competing implement
Real-world viability: Built for the failure paths, not just the happy path. The runner records stop-point build failures per-row without killing the run (freelunch.ts:717-728), catches and records rollout errors (737-741), treats provider 503s as recorded model-error exits rather than dropping them (doc reports 3 such rollouts kept in the denominator), bounds concurrency via mapLimit, and handles the real Docker constr
Model: opencode/zai-coding-plan/glm-5.2
Bridge attempts: 1

🔎 Heuristic Signals

🟡 Cruft: magic number added scripts/tb-repair-freelunch.ts

 const seconds = Math.ceil(timeoutMs / 1000)

🎯 Usefulness Audit

🟡 New load-bearing runner scripts sit outside the typecheck net [robustness] ``

tsconfig.script.json:4 includes only scripts/tb-repair-milestone1.ts, so pnpm typecheck:scripts does not cover tb-repair-freelunch.ts, tb-repair-freelunch-model.ts, or tb-oracle-determinism.ts — the three files this measurement now depends on. This matches the repo's existing convention (many scripts are unlisted), so it is not a deviation, but these scripts are now the canonical artifact for a decision-relevant number. Adding them to tsconfig.script.json's include array would cost nothing and

💰 Value Audit

🟡 Campaign runner helpers duplicated between tb-repair scripts [duplication] ``

Five helpers (stepTimeoutMs, recordedReturncode, loadSuite, taskOracle, mapLimit) are copied verbatim between scripts/tb-repair-freelunch.ts:126-198,531 and scripts/tb-repair-milestone1.ts:104-188,313,552. mapLimit (a bounded async worker pool) is now triplicated — also at src/harness-optimizer.ts:182. A shared scripts/tb-repair-helpers.ts (or promoting mapLimit to src/) would cut the copy. Low urgency: these are standalone campaign runners with different container lifecycles, and the helpers ar

What this audit checks

It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.

Pass	What it asks
Heuristic	Vague title? Whitespace-only or cruft-bearing diff? (content signals only)
Duplication	Do added function/class names already exist elsewhere in the repo?
Value Audit	What does it do? What goal does it achieve? Is it good? Better architecture or already-exists?
Usefulness Audit	Does it integrate and fit? Will it hold up in real use and actually get used?

Findings are concerns, not blocks — the human reviewer decides what to do with them.

_{value-audit · 20260810T115049Z}

tangletools · 2026-08-10T12:25:41Z

⚠️ Review Incomplete — `f1f03f2d`

At least one required reviewer lane failed closed. No approval or request-changes review was published. This is a reviewer run failure, not a PR quality score.

Trigger a fresh review on the current PR head.

_{tangletools · 2026-08-10T12:25:39Z}

tangletools

✅ Auto-approved drewstone PR — `f1f03f2d`

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

_{tangletools · auto-approval · reason: drewstone_author · 2026-08-10T12:27:13Z}

tangletools

🟠 Value Audit — better-approach-exists


Verdict	better-approach-exists
Concerns	3 (1 medium-concern, 1 low, 1 weak-concern)
Heuristic	0.1s
Duplication	0.0s
Interrogation	300.0s (2 bridge agents)
Total	300.1s

💰 Value — better-approach-exists

A rigorous 'free-lunch' measurement that turns a structurally-null 0/96 control into a real, costed rescue rate — sound design, but its router model-call adapter reinvents the package's existing callLlm/assertServedModel layer.

What it does: Re-runs TB-Repair admission condition 3 ('is a failed row rescued by continuing unconditionally from its recorded end state?') with a real model budget. The two milestone runs had answered it under a zero-step control that makes no model call, so it graded the same failing bytes and returned 0/96 — the only number it could return. This adds: (1) two substrate modules — control-policy.ts (a declare
Goals it achieves: Turn a measurement that was structurally forced to zero into a real one, so admission condition 3 is calibrated against an actual screen rate and a per-rollout cost — which licenses (without proving) a downstream paired gate-vs-no-gate study that can budget-match against the $0.148/rollout and 4.7% figures. Secondary: stop letting a control that cannot act silently pass every row, and stop letting
Assessment: The measurement methodology is coherent and unusually disciplined for a one-off campaign: isolation is proven rather than asserted (DNS/TCP fail inside the continuation container, succeed in a networked control from the same image), the stop point is reconstructed once per row so within-row variance is only the continuation, the served model is verified against the requested id, model-error rollou
Better / existing approach: The router model-call adapter reinvents infrastructure the package already centralizes. scripts/tb-repair-freelunch-model.ts reimplements, from raw fetch: the retry loop (post, lines 117-157), the served-model substitution guard (servedMatches + ModelSubstitutionError, lines 65-68 and 160-184), live pricing from /v1/models (fetchPricing, lines 77-97), and usage extraction (toRunTokenUsage, lines 9
Model: opencode/zai-coding-plan/glm-5.2
Bridge attempts: 2
Bridge warning: opencode/kimi-for-coding/k2p7: opencode: opencode error

🎯 Usefulness — sound

Adds the free-lunch measurement runner — unconditional continuation with a real model budget — which has already been exercised to produce a measured 4.7% rescue rate, and fits the existing campaign-runner pattern exactly.

Integration: Fully wired and already exercised. The main runner scripts/tb-repair-freelunch.ts consumes substrate modules exported from src/trace-repair/index.ts (continuation-policy, oracle-determinism, control-policy, injectedTestOracle). It is invoked via npx tsx and documented with reproduction commands in docs/trace-repair-free-lunch.md:97-104. The oracle CLI scripts/tb-oracle-determinism.ts is
Fit with existing patterns: Follows the established pattern precisely. tb-repair-milestone1.ts already established the campaign-runner-consumes-substrate shape; tb-repair-freelunch.ts extends it from a zero-step control to a real continuation budget. The three-container-generation design (replay networked → continue --network none → grade networked) is a direct response to Docker's refusal to attach a network to a none
Real-world viability: Handles realistic conditions throughout. Stop-point caching across rollout passes (buildStopPoint with record-file reuse) matches the incremental-campaign pattern. Slug collision detection (tb-repair-freelunch.ts:613) prevents two rows silently sharing a container state. Error paths are recorded not dropped: model 503/429 retried then recorded as model-error, substitution detected and stops
Model: opencode/zai-coding-plan/glm-5.2
Bridge attempts: 1

🔎 Heuristic Signals

🟡 Cruft: magic number added scripts/tb-repair-freelunch.ts

 const seconds = Math.ceil(timeoutMs / 1000)

💰 Value Audit

🟠 Model adapter reimplements callLlm + assertServedModel instead of wrapping them [duplication] ``

scripts/tb-repair-freelunch-model.ts:117-184 (post retry loop, servedMatches, ModelSubstitutionError, fetchPricing) duplicates src/llm-client.ts:632-857 (callLlm retry + assertServedModelIdentity at 832-838 + costUsd at 852) and src/integrity/served-model.ts:111-156 (checkServedModel). Commit 4a51f08 re-derived the exact substitution bug (router serving v4-flash for v3.2) that the shared assertServedModel was built to catch — a signal the wheel is being reinvented. Better: wrap callLlm (or creat

🟡 ~150 lines of docker/suite helpers copy-pasted between the two campaign runners [maintenance] ``

scripts/tb-repair-freelunch.ts duplicates near-verbatim from scripts/tb-repair-milestone1.ts: taskOracle (freelunch:188-198 == milestone1:313-323), stepTimeoutMs (126-130 == 104-108), recordedReturncode (132-136 == 373-377), mapLimit (531-549 == 552-570), loadSuite (138-154 ~ 174-188), and the docker exec helper (dockerSession/startNetworkedContainer 201-261 ~ dockerSessions 233-304). These are one-off campaign runners so this is not load-bearing, but a shared scripts/tb-repair-harness.ts (or pr

What this audit checks

It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.

Pass	What it asks
Heuristic	Vague title? Whitespace-only or cruft-bearing diff? (content signals only)
Duplication	Do added function/class names already exist elsewhere in the repo?
Value Audit	What does it do? What goal does it achieve? Is it good? Better architecture or already-exists?
Usefulness Audit	Does it integrate and fit? Will it hold up in real use and actually get used?

Findings are concerns, not blocks — the human reviewer decides what to do with them.

_{value-audit · 20260810T131337Z}

tangletools · 2026-08-10T13:48:55Z

❌ Needs Work — `f1f03f2d`

Review health 100/100 · Reviewer score 0/100 · Confidence 95/100 · 78 findings (1 high, 24 medium, 53 low)

	opencode GLM 5.2	opencode DeepSeek v4 Pro	opencode DeepSeek v4 Flash	aggregate
Readiness	0	0	0	0
Confidence	95	95	95	95
Correctness	0	0	0	0
Security	0	0	0	0
Testing	0	0	0	0
Architecture	0	0	0	0

Reviewer score is advisory once the run is complete and the verdict has no blockers.

Full multi-shot audit completed 8/8 planned shots over 30 changed files. Global verifier still owns final merge decision. | Full multi-shot audit completed 8/8 planned shots over 30 changed files. Global verifier still owns final merge decision. | Full multi-shot audit completed 8/8 planned shots over 30 changed files. Global verifier still owns final merge decision.

Blocking

🔴 HIGH ROLLOUT_BASE seed shift is a no-op; every pass draws the index-0 seed — scripts/tb-repair-freelunch.ts

Lines 63-71 document that a campaign runs uniform passes and shifts ROLLOUT_BASE so pass 2 'draws the seeds of index 1 rather than redrawing index 0', and that 'an index that repeats is a rollout that repeats'. That mechanism does not reach the seed: runRollout calls runContinuation once per external rollout index with rollouts: 1 (line 439), and src/trace-repair/continuation-policy.ts:240-256+286 derives the per-rollout seed as continuationSeed(policy.seed, rowId, index) from its own internal loop counter, which is always 0 when

Other

🟠 MEDIUM Replicate re-grades read a stale reward file; a failing run is recorded as a pass — benchmarks/trace-repair/tools/assay-tbench-corpus.mjs

The new re-grade loop runs /tests/test.sh and then readReward() without clearing /logs/verifier/reward.txt between replicates (lines 524-538). If a re-grade fails before test.sh writes the reward, cat succeeds on the previous replicate's file and the failing run is recorded as a pass (reward '1'). The shell counterpart does this correctly: replicate_group runs rm -f /logs/verifier/reward.txt /logs/verifier/ctrf.json before every run (certify-task-oracle.sh:254) and run_tests does the same ([line 192](https://github.com/tangle-network/agent-eval/blob/f1f03f2da32103d3619fb3f3b26b4942de9b4034/benchmarks/trace-repair/tools/a

🟠 MEDIUM Stale reward.txt read between re-grades in assay verify-task — benchmarks/trace-repair/tools/assay-tbench-corpus.mjs

The re-grade loop (lines 524-537) does not remove /logs/verifier/reward.txt before each replicate. If test.sh crashes before writing reward.txt, the previous run's stale reward is read as the current run's result. The shell script's replicate_group explicitly guards against this at certify-task-oracle.sh:254 with docker exec "$1" rm -f /logs/verifier/reward.txt. Fix: add sh(['exec', cid, 'rm', '-f', '/logs/verifier/reward.txt']) at the start of each loop iteration.

🟠 MEDIUM DETERMINISM_RULE_UNAVAILABLE leaks both phase C containers — benchmarks/trace-repair/tools/certify-task-oracle.sh

When the substrate rule cannot be applied, case $? in 0|3) ;; *) ... return 1 returns before the docker rm -f "$ca" "$cb" cleanup at lines 422-423. Both tb-certify-a-* and tb-certify-b-* stay running (plus any busy-loops start_load detached), until an operator removes them; a later run's start_container does docker rm -f first so the leak is masked until then, but it leaves load on the host. Fix: remove the containers before the early return, or add a trap that cleans up both names.

🟠 MEDIUM Docker container leak on Phase B / Phase C failure paths — benchmarks/trace-repair/tools/certify-task-oracle.sh

Container 'ca' (phase A) was kept alive for Phase C (line 353 comment) but cleanup was moved to line 422. Three early-return paths now leak 'ca': (a) line 358 — start_container for 'cb' fails, (b) [line 379](https://github.com/tangle-network/agent-eval/blob/f1f03f2da32103d3619fb3f3b26b4942de9b4034/benchmarks/trace-repair/tools/certify-task-or

🟠 MEDIUM Exit code 2 conflates file-I/O errors with oracleDeterminism validation rejections — scripts/tb-oracle-determinism.ts

The fail function always exits with code 2, called from both the file-read catch (line 37) and the oracleDeterminism catch (line 44). The shell caller certify-task-oracle.sh at line 404 treats only exit 0 and 3 as 'the rule ran' and any other exit as 'DETERMINISM_RULE_UNAVAILABLE'. When oracleDeterminism throws a ValidationError (empty groups, insufficient replicates), exit 2 tells the operato

🟠 MEDIUM The wrapper's exit-code and stdout contract is untested — scripts/tb-oracle-determinism.ts

certify-task-oracle.test.mjs covers only the shell's --self-test and flag validation; nothing invokes the tsx script, so the mapping exit 0/3/2 and the exact flip_bp=.. replicates=.. state=.. line the shell's case parser depends on (certify-task-oracle.sh:411-417) can drift without any test failing. The substrate rule is well tested (12/12) but the serialization boundary between rule and shell is not. Fix: add a test that runs the tsx script against a stable and a flipping fixture evidence file and asserts stdout fields and exit codes.

🟠 MEDIUM Non-retryable 4xx errors retried after throw is caught by generic catch — scripts/tb-repair-freelunch-model.ts

Line 145 throws on non-retryable HTTP errors (status < 500, not 429), but the throw occurs inside the try block and is caught by the generic catch at line 148. The catch block treats all errors as retryable — it waits (attempt * 5000ms) and continues the loop. A 400/401/403/404 error will retry up to MAX_ATTEMPTS (4) times before falling through to line 156, wasting ~30s of waiting and 4x

🟠 MEDIUM Pricing unit is undocumented and unverified; silent cost corruption if wrong — scripts/tb-repair-freelunch-model.ts

fetchPricing (line 77) reads pricing.prompt/pricing.completion as strings from /v1/models and Number()s them; the cost calc at line 200 (usage.input * pricing.prompt + usage.output * pricing.completion) treats the result as USD-per-token. The ModelPricing doc at line 29 asserts 'USD per prompt token' but nothing verifies the router's actual unit. The repo itself is inconsistent: src/me

🟠 MEDIUM Unpriced model throws instead of returning costUsd: null as documented — scripts/tb-repair-freelunch-model.ts

Header (lines 8-10) promises: 'a model with no local price returns costUsd: null, which makes the whole rollout's cost uncaptured rather than a sum that is too small'. Implementation contradicts it: fetchPricing throws at line 95 when the router publishes no per-token rate, and routerContinuationModel calls it unguarded at line 186 AFTER a successful (paid) chat completion. The throw prop

🟠 MEDIUM fetchPricing exact-ID lookup mismatches servedMatches tail-match — scripts/tb-repair-freelunch-model.ts

servedMatches (line 65-67) uses a tail-segment match (e.g. 'z-ai/glm-5.2' matches 'glm-5.2'), but fetchPricing does pricingCache.get(model) with the exact served ID. If the router serves a vendor-prefixed ID but the /v1/models endpoint only lists the bare ID with pricing, the lookup fails with 'router publishes no per-token price for '. The substitution check passes but pricing fails. The docstring on line 90 says 'Throws when it publishes none' but doesn't warn about the naming gap. Mitigation exists: the router

🟠 MEDIUM post() retries non-retryable 4xx errors, contradicting its own docstring — scripts/tb-repair-freelunch-model.ts

The docstring at line 112-116 says 'Retries on transport failure and on the router's own 5xx'. The code does the opposite for 4xx: inside the !response.ok branch, non-5xx/non-429 statuses execute throw new Error(lastError) (line 145), but that throw is caught by the surrounding catch (error) (line 148), which sets lastError and — unless attempt===MAX_ATTEMPTS — sleeps attempt*5_00

🟠 MEDIUM servedMatches duplicates and weakens the repo's served-model integrity check — scripts/tb-repair-freelunch-model.ts

The same PR adds src/integrity/served-model.ts with normalizeModelId/checkServedModel that strip provider prefix, @snapshot, :batch/:free tier, and trailing build dates (lines 70-99), and its comment documents that the router answers with aliased forms (e.g. 'gpt-4o-mini-2024-07-18'). This file instead compares only the final '/'-segment lowercased (line 66). Consequence: a legitimate alias response ('glm-5.2' served as 'z-ai/glm-5.2:free' or 'glm-5.2-2026-01-15') fails servedMatches at [line 180](https://github.com/tang

🟠 MEDIUM New 791-line script is not covered by pnpm typecheck:scripts — scripts/tb-repair-freelunch.ts

The script imports 18 symbols from ../src/trace-repair and one from ./tb-repair-freelunch-model. CI runs pnpm typecheck:scripts (ci.yml:47) which uses tsconfig.script.json, whose include lists only scripts/tb-repair-milestone1.ts. Neither tb-repair-freelunch.ts nor tb-repair-freelunch-model.ts is in that list, so any future refactor of trace-repair exports (e.g. renaming ContinuationRollout, changing OracleDeterminismVerdict.stable, or moving RecordedTrajectoryStep) will compile in CI while silently breaking this script. I confirmed the file currently typechecks clean by constructing a temporary tsconfig that includes it with the real src/ tree (0 errors, TS 7.0.2), so this is a coverage gap, not a present defect. Impact: a future substrate change could land green and break the campaig

🟠 MEDIUM ROLLOUT_BASE has no effect on seeding; all rollouts share seed index 0 — scripts/tb-repair-freelunch.ts

Lines 62-68 document ROLLOUT_BASE: 'The seed derives from the row and the index, so an index that repeats is a rollout that repeats.' But runRollout calls runContinuation with rollouts: 1 at line 441, so the internal index in runContinuation is always 0 and the seed is always continuationSeed(policy.seed, rowId, 0) (src/trace-repair/continuation-policy.ts:286). The rolloutIndex parameter flows into logging and outcome records but never reaches continuationSeed. At T=0 this makes ROLLOUTS > 1 duplicate identical rollou

🟠 MEDIUM Slug collision guard is case-sensitive, stop-point image identity is not — scripts/tb-repair-freelunch.ts

slug() (lines 334-336) preserves case, but the stop-point image tag lowercases it: tbfl/stop:${base} where base = tbfl-${slug(row.rowId)}.toLowerCase() (lines 374-375). The pre-build guard (lines 613-621) keys on the case-sensitive slug, so two rowIds that differ only by case (e.g. t__AbC vs t__aBC) pass the guard yet share one image tag while writing distinct record files (stopPointRecor

🟠 MEDIUM Certification suite digest is never pinned to the suite the run grades — scripts/tb-repair-milestone1.ts

Evidence: loadTask() computes suiteDigest = testSuiteDigest(suite) from the live TB2 clone (line 226) and loadTaskOracles() parses the certification's own suiteDigest (line 150); nothing compares them, and admitRow (admission-contract.ts) never compares evidence.oracleDeterminism.suiteDigest to evidence.suiteDigest either. The new docstring claims 'every check below reads that suite', but a drifted clone (certify-task-oracle.sh explicitly warns 'certification is a decaying asset' and 'Pin the clone. Task definitions move under uncha

🟠 MEDIUM Unstable oracle tasks waste containers before guaranteed rejection — scripts/tb-repair-milestone1.ts

The pre-filter at line 580 only rejects uncertified tasks (!taskOracles.has(name)), not tasks with stable: false. Tasks with OracleDeterminismVerdict.stable === false pass this filter but every row is always rejected by admitRow at admission-contract.ts:226 with 'task-oracle-nondeterministic'. The rejection happens AFTER processRow runs all control rollouts (3 noFix + 3 noOp = 6 docker sessions per row). The PR comment at line 31-33 says such tasks 'does not run', but they do — they just always get rejected. Add `!tas

🟠 MEDIUM Unstable-certified task is not halted before containers open — scripts/tb-repair-milestone1.ts

Evidence: the gate filters only on taskOracles.has(name); largest-eigenval is present in the checked-in task-oracles.json with stable=false and flipRate 0.375 (verified by running parseTaskOracleRegistry against the file: 'unsolved: 0/8 suite pass, per-assertion counting, 8 unit(s) flipped, worst test_speedup[10] 5/8 pass'). The run therefore proceeds: processRow opens 6 containers and runs 6 full trajectory replays per largest-eigenval row (lines 441-471) BEFORE admitRow rejects the row as task-oracle-nondeterministic (the contract's determinism check would have been first only if the script used runAdmission's ordering). The docstring says such a task 'd

🟠 MEDIUM controlScreening:'declared-inert' is unreachable on the executed admission path — src/trace-repair/admission.ts

resolveAdmissionConfig/ADMISSION_CONFIG_DEFAULTS advertise 'declared-inert' (admission.ts:200-204), but assertControlCalibrated requires stepBudget===0 for that mode while definePinnedContinuationPolicy enforces stepBudget>=1 (continuation-policy.ts:98). Probe: runAdmission({config:{controlScreening:'declared-inert'}, policy: definePinnedContinuationPolicy(...)}) always throws UncalibratedControlError, so the option is dead through the public API and can only run with a hand-built policy object that bypasses the constructor. Additionally, were it reached, a control pass is recorded as 'no-fix-control-rescued' ('the continuation policy passes the task with no intervention') in the executed path, while the contract path records the same event as 'control-passed-on-identical-state' — two voca

🟠 MEDIUM defineControlPolicy validation branches have no direct unit tests — src/trace-repair/control-policy.ts

defineControlPolicy has validation branches for model/stepBudget co-consistency (zero budget with non-null model at line 107, positive budget with no model at line 113) that are never tested directly. assertControlCalibrated is tested indirectly via admission.test.ts (refuses to screen under a control that makes no model call), but defineControlPolicy validation errors would surface only when a caller misconfigures the policy factory — a different failure mode than the admission pre-check.

🟠 MEDIUM No dedicated tests for oracle-determinism.ts or control-policy.ts — src/trace-repair/oracle-determinism.ts

Two new pure modules (330 + 165 lines) have zero dedicated test files (confirmed: only admission.test.ts, admission-report.test.ts, continuation-policy.test.ts, docker-environment.test.ts, mini-swe-scaffold.test.ts exist). Complex reduction logic is only exercised indirectly via the certification() helper in admission.test.ts which always uses assertions:null and a single idle group. Untested paths: per-assertion vs reward granularity switching, assertionSetUnstable detection (lines 214-215), loadSensitive computation across idle+contended groups ([line 252](https://github.com/tangle-network/agent-eval/blob/f1f03f2da32103d3619fb3f3b26b4942de9b4034/s

🟠 MEDIUM Per-assertion determinism mode ignores whole-suite reward flips, so a flaky grader can be certified stable — src/trace-repair/oracle-determinism.ts

In the perAssertion branch the reducer counts flips only among the reported assertions and never examines replicate.passed/reward, even though it computes the state's passes/fails from them (line 193). Probe: two replicates both report assertion {id:'t',passed:true} but passed=true/reward='1' vs passed=false/reward='0' -> verdict { stable:true, flipRate:0, passes:1, fails:1, rewardsObserved:['0','1'] }. The module's premise 'a pass/fail reward is a conjunction over many assertions' (lines 16-23) is asserted,

🟠 MEDIUM Per-assertion oracle-determinism counting path has zero test coverage — src/trace-repair/oracle-determinism.ts

The per-assertion branch (lines 205-230) — assertion set stability detection, per-unit minority counting, absent-replicate-as-failure aggregation — is never exercised by any test. The admission.test.ts certification() helper always passes assertions: null, taking the reward-only code path (lines 231-240). The assertion-level logic that detects flips at finer granularity is the primary motivation for the per-assertion design (see comment [lines 16-23](https://github.com/tangle-network/agent-eval/blob/f1f03

🟠 MEDIUM parseTaskOracleRegistry does not validate nested JSON structure — src/trace-repair/oracle-determinism.ts

The function takes document: unknown but only validates top-level shape (version === 1, measurements is array). It casts via as Partial<TaskOracleRegistryDocument> then maps each measurement through oracleDeterminism without checking that groups, replicates, assertions, passed, wallMs, etc. exist and have correct types. If a checked-in JSON certification file is malformed (e.g., missing groups array, replicates as string, passed as number), oracleDeterminism will throw an unstructured TypeError (e.g., 'Cannot read properties of undefined') rather than a ValidationError. Worse: a group with an unrecognized state value (e.g., 'partial') would be silently dropped by the ['unsolved','solved'] filter at [line 167-169](https://github.com/tangle-network/agent-eval/blob/f1f03f2da32103d3619fb3f

🟡 LOW Checked-in registry contains a task the determinism rule marks unstable, with no invariant test — benchmarks/trace-repair/task-oracles.json

parseTaskOracleRegistry on the shipped file returns largest-eigenval with stable=false, flipRate=0.375, detail 'unsolved: 8 unit(s) flipped, worst test_outputs.py::test_speedup[10] 5/8 pass'. Direct recount of the unsolved/idle replicates confirms 8 parametrized test_speedup[N] timing assertions return different verdicts on byte-identical state (e.g. test_speedup[2] 1/5 pass, test_speedup[3] 4/5 pass). This is exactly the per-assertion nondeterminism phase C exists to detect. Functionally safe: admission.ts:419 excludes it as 'task-oracle-nondeterministic', tb-repair-freelunch.ts:560 filters .stable===true, tb-repair-milestone1.ts feeds the admission contract. BUT tools/README.md:160 says 'Copy a task\u2019s determinism.json into the measurements array to certify it', implying membership =

🟡 LOW task-oracles.json holds a task whose replicates are not stable — benchmarks/trace-repair/task-oracles.json

The largest-eigenval measurement (line 229) re-derives through parseTaskOracleRegistry to stable:false, flipRate 0.375: test_speedup[*] timing assertions flip on the unsolved state under both idle and contended load, while the suite reward stays 0 (conjunction hides it). This is correct behavior and the data is internally consistent (verified), and every consumer checks .stable (admission.ts:419, tb-repair-freelunch.ts:560), so the task is excluded as task-oracle-nondeterministic, not certified. But README's task-oracles.json section ('Copy a task's determinism.json into the measurements array to certify it') implies the file holds only certified ta

🟡 LOW Assay regrade loop does not clear reward.txt between replicates, unlike the shell certifier — benchmarks/trace-repair/tools/assay-tbench-corpus.mjs

The new regrade loop (lines 523-538) calls readReward() each iteration but never removes /logs/verifier/reward.txt before re-running test.sh. certify-task-oracle.sh clears it before every grade (run_tests line 192, replicate_group line 254) precisely so a stale file cannot be read as this run's verdict. Here, replicate 0 inherits the

🟡 LOW Contention load outlives its group and can contaminate the second state's idle group — benchmarks/trace-repair/tools/certify-task-oracle.sh

bound=$(( (${4:-30} * (DETERMINISM_LOAD + 1)) + 30 )) sizes the busy-loop lifetime from the other state's phase grade time, and the loops run inside a container whose CPU share is host-wide. Unsolved contended loops (alive ~grade+30s after that group ends) can still be spinning when the solved state's idle group starts, so the 'idle' label is not what ran. Direction is fail-safe: the substrate pools loads per state, so residual load only adds flips and rejects, never falsely certifies. Also, if a contended replicate hits verifier_timeout the loops expire mid-group and the remaining replicates run uncontended. Fix: bound the loops to the group (e.g. start them per-replicate, or compute bound from the state's own projected duration and tear them down after the group).

🟡 LOW Temp file leak on npx tsx failure — benchmarks/trace-repair/tools/certify-task-oracle.sh

groups_file (created by mktemp at line 386) is cleaned at line 420, but the early return at line 406 (npx tsx exit code not 0 or 3) skips that cleanup. Fix: move rm -f "$groups_file" into a trap or duplicate it on the error path.

🟡 LOW summary.psv header changed 12 -> 14 columns but is only written when the file is absent — benchmarks/trace-repair/tools/certify-task-oracle.sh

Line 162 writes the header only when $SUMMARY is empty ('[ -s "$SUMMARY" ] || echo ... > "$SUMMARY"'), and the header grew two columns (determinism_replicates, determinism_flip_bp). An existing summary.psv from a prior certifier run keeps its 12-column header while new rows append with 14 columns, producing a mixed-format PSV that any '|'-split parser misreads. Low blast radius (operational tool, old summaries are regenerable), but a header-version guard or an unconditional rewrite on schema change would prevent silent misalignment.

🟡 LOW summary.psv schema drift on pre-existing files — benchmarks/trace-repair/tools/certify-task-oracle.sh

The header is written only when the file is absent ([ -s "$SUMMARY" ] || echo ...). An operator with a summary.psv from before this PR (11 columns) will get 13-column rows appended (lines 428), silently misaligning determinism_replicates/determinism_flip_bp/verdict under old column names. Fix: detect a stale header (column-count mismatch) and write a new header line, or version the file name.

🟡 LOW "rate" column in by-task table is rollout-rate, ambiguous next to a "rows" column — docs/trace-repair-free-lunch.md

The by-task table (lines 41-45) lists both rows and rollouts columns, then a single rate column whose values are passes/rollouts (10.0%, 8.3%, 0.0%). A reader scanning the table after the section header — which leads with the row-level 6.2% — could misread 10.0% as a row-level rate. No factual error; rename to rollout rate or add a second row rate column for parity with the headline.

🟡 LOW 'after four retries' overstates retry count — docs/trace-repair-free-lunch.md

The model-error rollouts are described as ended 'after four retries', but scripts/tb-repair-freelunch-model.ts sets MAX_ATTEMPTS=4, which is four total attempts (three retries); post() throws 'router call failed after 4 attempts'. Wording suggests four retries on top of an initial call. Fix: say 'after four attempts'. No numerical impact; the rate tables are unaffected.

🟡 LOW 'same verdict in all 16 certification replicates' is loose phrasing — docs/trace-repair-free-lunch.md

benchmarks/trace-repair/task-oracles.json shows count-dataset-tokens graded with 16 replicates split across states: unsolved always fails (8/8), solved always passes (8/8). The suite does not return one identical verdict across all 16 replicates; it is deterministic per state (flipRate 0, stable). The intended meaning (the rescue is not a flaky-grader flip) is correct and verified, but the phrasing could be read as 'all 16 replicates returned the same verdict'. Consider 'graded the same state identically in all 16 certification replicates'.

🟡 LOW 3-cluster task bootstrap interval is acknowledged-but-uninformative — docs/trace-repair-free-lunch.md

The task-clustered bootstrap CI (0.0%-10.0%) over 3 clusters is reported and then dismissed in the same breath (line 22 and Threat 3). It adds no information a reader can use; the row-clustered and Clopper-Pearson intervals already cover it. Keeping it is defensible for completeness, but a footnote-style treatment would tighten the section.

🟡 LOW --out value before the evidence path is misread as the evidence file — scripts/tb-oracle-determinism.ts

evidencePath = args.find((arg) => !arg.startsWith('--')) takes the first non-flag argument, so node ... tb-oracle-determinism.ts --out verdict.json evidence.json picks verdict.json as the evidence path. Verified: exit 2 with 'cannot read : ENOENT'. Not position-independent despite the documented form EVIDENCE.json [--out VERDICT.json]. Only in-repo caller (certify-task-oracle.sh:402) passes evidence first, so latent. Fix: consume --out and its value first, then treat the remaining positional as evidence; or error when a second positional appears.

🟡 LOW --out with missing path silently produces no verdict file — scripts/tb-oracle-determinism.ts

const outPath = outIndex >= 0 ? args[outIndex + 1] : null yields undefined when --out is the last argument with no following path. Line 47 if (outPath) silently skips the write. The current caller always passes a value, but a future caller that omits it gets no diagnostic. Fix: guard outIndex + 1 >= args.length and fail with a clear message before proceeding.

🟡 LOW Header doc conflates 'cannot read evidence' with 'rule rejected evidence' — scripts/tb-oracle-determinism.ts

Lines 11-13: 'Exit 0 when the grader answered about the state, 3 when it did not, 2 when the evidence could not be read.' But fail() is also called from the rule-application catch on line 44, which fires when oracleDeterminism throws ValidationError (e.g., a group with <2 replicates). That is not a read failure; it is the rule refusing the evidence. The bash caller does not distinguish (any non-{0,3} is 'unavailable'), so behavior is correct; only the comment is misleading. Suggest: 'Exit 2 when the evidence cannot be read

🟡 LOW Malformed evidence can silently yield a 'stable' verdict (no shape validation at the script boundary) — scripts/tb-oracle-determinism.ts

The script JSON.parses and casts to OracleDeterminismEvidence with no structural validation, and the substrate's oracleDeterminism skips any group whose state is not 'unsolved'/'solved' without complaint. A typo'd or unexpected state label in evidence would drop that group's replicates from the verdict and could report stable=flipRate 0 over a subset of the measured data. Evidence is machine-generated by the shell with fixed labels today, so live risk is low, but this contradicts the repo's 'fail loud' doctrine. Root cause is out-of-shot (src/trace-repair/oracle-determinism.ts); the script could add a cheap group-count/state-label sanity check at the entry point. Noting for the global verifier.

🟡 LOW No direct unit tests for the CLI wrapper script — scripts/tb-oracle-determinism.ts

The substrate function oracleDeterminism has 200 lines of tests in tests/trace-repair/oracle-determinism.test.ts covering every counting mode, invalid-input rejection, load-sensitivity, and the registry. This 63-line CLI wrapper has none. It is exercised indirectly by certify-task-oracle.test.mjs --self-test, but that path does not verify exit-code semantics, stderr format, or the --out file contents. A direct test would have caught finding #1 (exit-code conflation) and finding #2 (missing --out value).

🟡 LOW --out with no value is silently ignored — scripts/tb-oracle-determinism.ts

Line 30-31: const outIndex = args.indexOf('--out'); const outPath = outIndex >= 0 ? args[outIndex + 1] : null. When --out is the last token, args[outIndex + 1] is undefined, which is falsy, so the if (outPath) guard on line 47 skips the write with no warning. The verdict still prints to stdout and the exit code is unchanged, so the caller is unaffected, but a human running the script directly loses the verdict file silently. Fix: treat outPath === undefined after the lookup as a usage error (`fail('--out needs a path

🟡 LOW process.exit() can truncate the stdout verdict line the shell parses — scripts/tb-oracle-determinism.ts

Node's process.exit() terminates without draining async writes to process.stdout when it is a pipe. certify-task-oracle.sh captures stdout via fields=$(...) (line 402-410) and derives the verdict only from those fields, defaulting det_state=skipped, det_flip_bp=0, det_replicates=0 when the line is empty; certify_verdict then records CERTIFIED_UNCHECKED_DETERMINISM for a 0/1 suite even when the rule exited 3 as NONDETERMINISTIC. A silent wrong certification in eval-validity plumbing. Probability is low (output is ~60 bytes) but the consequence is a coin-flip grader shipped as 'unchecked'. Fix: set process.exitCode and let the loop drain, or call process.

🟡 LOW writeFileSync not guarded; write failure exits 1 instead of 2 — scripts/tb-oracle-determinism.ts

Line 47: if (outPath) writeFileSync(outPath, ...) is outside any try/catch. A write failure (missing parent dir, ENOSPC, EACCES) throws and Node exits 1. The bash caller's case $? in 0|3) ;; *) ...DETERMINISM_RULE_UNAVAILABLE (certify-task-oracle.sh:404) still treats this as rule-unavailable — fail-closed — but the human stderr summary and the stdout shell-fields are both suppressed because the crash happens before lines 50-62 run. Wrap the write in try/catch that routes to fail() (exit 2) for consistency with the read pat

🟡 LOW Cached prompt tokens charged at full rate; cost overstated when caching is active — scripts/tb-repair-freelunch-model.ts

toRunTokenUsage (line 107) captures prompt_tokens_details.cached_tokens into RunTokenUsage.cached, but the cost calc at line 200 multiplies the entire usage.input (which includes the cached subset per OpenAI/OpenRouter convention) by the full pricing.prompt rate. If the router applies a cache discount (typically 50-100% off), recorded cost is systematically high. The top-of-file doc ([lines 5-14](https://github.com/tangle-network/agent-eval/blob/f1f03f2da32103d3619fb3f3b26b4942de9b4034/scripts/tb-repair-freelunch-mod

🟡 LOW No test coverage for the freelunch model script — scripts/tb-repair-freelunch-model.ts

No dedicated test file found (globbed */freelunchtest returned nothing). The script contains retry logic, pricing cache population, token-usage conversion, model-substitution detection, and cost arithmetic — all untested. As a script under scripts/ the testing bar is lower than for library code, but the cost calculation and retry policy have enough surface area to benefit from a unit test.

🟡 LOW No unit tests for exported measurement-critical functions — scripts/tb-repair-freelunch-model.ts

servedMatches and fetchPricing are exported but nothing imports them in a test (grep across scripts/, tests/, src/ finds only production callers; the file has no .test.ts sibling). The alias/usage-null/cost-null rules this file enforces are exactly the integrity semantics that cause silent mis-measurement if wrong, and toRunTokenUsage (line 99), the 429/5xx retry path, and the pricing parse (Number() of router strings, line 88-89) are untested. Fix: add unit tests for servedMatches against the alias forms served-model.ts

🟡 LOW Number(envVar) can silently yield NaN with a confusing diagnostic — scripts/tb-repair-freelunch-model.ts

Lines 25-26: Number(process.env.TBR_FL_TIMEOUT_MS ?? '180000') and Number(process.env.TBR_FL_ATTEMPTS ?? '4'). A non-numeric env value (e.g., TBR_FL_ATTEMPTS=auto) produces NaN: 1 <= NaN is false, so the post() loop body never executes and the function throws router call failed after NaN attempts: — a misleading message that hides the real cause (bad env). setTimeout(fn, NaN) is treated as 0, so a bad timeout silently disables the timeout. Add Number.isFinite validation with a thrown error naming the offending env var.

🟡 LOW post() retries non-retryable 4xx errors despite its documented retry policy — scripts/tb-repair-freelunch-model.ts

The comment (lines 112-115) states retries cover 'transport failure and the router's own 5xx'. But the throw for a non-retryable status (line 145) is inside the try block and is caught by the catch at line 148, which retries: a 400/401/404/422 is re-sent up to MAX_ATTEMPTS times with 5/10/15s backoff. A bad TANGLE_API_KEY (401) or malformed request therefore takes ~30s and 4 full payl

🟡 LOW Concurrent writeFileSync to rollouts.jsonl may interleave large lines — scripts/tb-repair-freelunch.ts

Multiple rollouts run concurrently via Promise.all (line 731) and each calls writeFileSync(join(WORK, 'rollouts.jsonl'), ..., { flag: 'a' }) at line 518. O_APPEND guarantees atomicity only for writes up to PIPE_BUF (typically 4096 bytes). If a single outcome JSON line exceeds that, concurrent writes may interleave, producing a corrupted line. Risk is low for typical outcome sizes (~500-2000 bytes). Mitigation: serialize writes through a queue, or use a per-worker output file and merge post-campaign.

🟡 LOW Crash between docker commit and record write orphans a stop-point image — scripts/tb-repair-freelunch.ts

buildStopPoint commits the Docker image at line 404 and writes the record file at line 407. A crash between these lines (e.g., OOM during writeFileSync) leaves the image on disk but no corresponding record. On restart, the image exists (line 396 check passes) but readFileSync(recordPath) at [line 399](https://github.com/tangle-network/agent-eval/blob/f1f03f2da32103d3619fb3f3b26b4942de9b4034/scripts/tb

🟡 LOW Helpers duplicate existing exports (parseRecordedReturncode, isRecordedTimeout) — scripts/tb-repair-freelunch.ts

recordedReturncode (lines 132-136) uses the identical regex /<returncode>(-?d+)<\/returncode>/ already exported as parseRecordedReturncode from src/trajectory-replay/steps.ts:32. stepTimeoutMs (lines 126-130) reimplements the substring check already exported as isRecordedTimeout from src/trace-repair/mini-swe-scaffold.ts:201 (the marker 'timed out and has been killed' is even defined as TIMEOUT_OBSERVATION_MARKER there). Impact: two sources of truth for the same parser; a future change to the returncode grammar or timeout

🟡 LOW No rebuild path when a stop-point image exists but its record file does not — scripts/tb-repair-freelunch.ts

The reuse branch (lines 378-392) treats the image's existence as authoritative and unconditionally reads stopPointRecordPath; if the record file is missing or corrupt (crash between docker commit on line 407 and writeFileSync on line 410, or manual cleanup), every row hard-fails as a stop-point error and there is no flag to force a rebuild. The resulting rows are recorded as errors for all ROLLOUTS (

🟡 LOW No tests for pure helpers and denominator-chain logic — scripts/tb-repair-freelunch.ts

The script has no tests. The pure helpers (slug, mapLimit, stepTimeoutMs, recordedReturncode) and especially the loadRows denominator-chain filter (lines 559-575: evaluated -> deterministicOracle -> cleanExit -> failedEndState -> primary/divergent, with the invariant failed.length === cleanExit.length) are unit-testable and subtle. A regression in the filter chain would silently change which rows run, biasing the measurement. Note: tb-repair-milestone1.ts has the same no-test property, so this is consistent with the existing campaign-script convention, but the gap is real. Fix: extract loadRows' filtering into a pure function taking {rows, admit, oracles}

🟡 LOW Prefix uses host uname, not the recorded run's system_information — scripts/tb-repair-freelunch.ts

Line 653-654 reads uname -a from the host running the replay and renders it into the continuation prefix via renderInstanceMessage({task, systemInformation}). The recorded agent originally saw its own host's uname, which may differ. The reconstructed prefix therefore is not byte-identical to what the recording's agent saw at decision time. Impact: probably immaterial for repair tasks (the model rarely keys off system_information for code fixes), and the script documents that assistant messages are elided so the prefix is reconstructed by design — but the uname choice is not explicitly justified. Worth a one-line comment noting that the host uname is a del

🟡 LOW Recorded-timeout steps replayed past the recording's killed state — scripts/tb-repair-freelunch.ts

For a step whose observation carries the 'timed out and has been killed' marker, stepTimeoutMs returns RECORDED_TIMEOUT_STEP_MS=60s (lines 82, 126-130) even though the recording's environment killed such commands at 30s. A long mutating command (e.g. apt-get/pip install) that the recording stopped at 30s is allowed to run to 60s during replay, so the committed stop point can hold state the recording never reached and the continuation then grades from a different state than the recorded end state. The comment on line 80-81 claims thi

🟡 LOW Zero test coverage for a 791-line measurement script — scripts/tb-repair-freelunch.ts

Neither scripts/tb-repair-freelunch.ts nor scripts/tb-repair-freelunch-model.ts has any test file. The script orchestrates Docker containers, router API calls, file I/O, and complex data transformations (replay, continuation, grading). A single smoke test (e.g., --plan output validation, set selection filter logic, slug collision detection) would catch the seed propagation issue and future regressions.

🟡 LOW dockerSession.exec misclassifies Node-level timeout as a normal failure — scripts/tb-repair-freelunch.ts

The catch block at lines 246-255 reads err.code and sets timedOut: exitCode === 124 || exitCode === 137. When the in-container timeout fires, docker exec exits 124/137 and the classification is correct. But the outer Node timeout: timeoutMs + 30_000 (line 243) fires an Error with signal: 'SIGTERM' and typically code: undefined when docker itself hangs; the fallback typeof err.code === 'number' ? err.code : 1 then yields exitCode=1 and timedOut=false. Impact: a hung docker exec is recorded as a normal exit-1 divergen

🟡 LOW routerSpend.deltaUsd labels total_usage as USD without confirmation — scripts/tb-repair-freelunch.ts

routerSpend() reads {total_usage, total_credits} from https://router.tangle.tools/v1/credits (lines 585-590) and returns them as {totalUsage, totalCredits}. The report then computes deltaUsd: spendAfter.totalUsage - spendBefore.totalUsage (line 778) and logs routerDelta=$... (line 785). The field name total_usage and the presence of a separate total_credits field (read but never used) suggest `

🟡 LOW Certification image provenance is loose for count-dataset-tokens — scripts/tb-repair-milestone1.ts

Evidence: task-oracles.json records image 'alexgshaw/count-dataset-tokens' with no @sha256: (archive-loaded at certify time), while the run's imageDigest() requires RepoDigests and the local image now carries one (verified: alexgshaw/count-dataset-tokens@sha256:9c513c...). No code compares certification.image to task.image (evidence.oracleDeterminism.image vs evidence.image), so this is the same unenforced-pin class as the suite digest. If a future run hits an image that is still archive-loaded, imageDigest() returns '' and docker run fails loudly; the mismatch is currently harmless but the audit trail is inconsistent.

🟡 LOW Header comment says unstable-oracle tasks 'do not run'; implementation rejects them per-row at admission — scripts/tb-repair-milestone1.ts

The block comment (lines 31-33) states a task whose suite returned different verdicts 'does not run'. main() only hard-stops on MISSING certifications (uncertified, line 580-586); UNSTABLE certifications (stable=false) still run their no-fix/no-op control containers during admission and are then rejected by admitRow with 'task-oracle-nondeterministic'. That is consistent with the substrate's soft-reject design and the rejection reason is recorded, but the prose overstates the gating. Tighten the wording to 'is rejected at ad

🟡 LOW INERT_CONTROL_CRITERIA.controlRollouts override is a no-op today — scripts/tb-repair-milestone1.ts

TB_REPAIR_ADMISSION_CRITERIA.controlRollouts is already 3 (admission-contract.ts:60) and CONTROL_ROLLOUTS is 3 (line 74), so the spread override sets the same value. This is defensible as a pin against future drift in the substrate default, but if that is the intent a one-line comment saying so would prevent a reader from 'cleaning it up' and silently re-coupling the runner to the substrate default. No behavior change required.

🟡 LOW Report artifact key renamed from 'policy' to 'controlPolicy' — scripts/tb-repair-milestone1.ts

The report object at line 610 changed from policy: ZERO_STEP_CONTINUATION_POLICY to controlPolicy: ZERO_STEP_CONTINUATION_POLICY. Any downstream consumer that reads the policy key from milestone1.json would get undefined. This is an internal benchmark script, so blast radius is limited, but the rename is a schema change without a migration note.

🟡 LOW Report carries certifications for tasks not in the sample — scripts/tb-repair-milestone1.ts

taskOracles is the entire parsed registry, so report.taskOracles serializes every certified task even when rows-sample.json uses a subset. Provenance is fine, but filtering to new Set(rows.map(r=>r.taskName)) would make the report self-describing (the certifications that actually screened these rows). Cosmetic; the four tasks currently in task-oracles.json make the cost invisible.

🟡 LOW deltaRepair pools rows across tasks and strata without explicit concatenation — scripts/tb-repair-milestone1.ts

Evidence: outcomes from all admitted rows across 4 tasks (and any stratum, since the script's criteria do not exclude signal-kill) are merged into one deltaRepair(graded, ...) call. admission.ts's own doctrine states pooling strata 'requires an explicit concatenation at the call site'. For a calibration run the 'control-cannot-rescue' and 'bootstrap-below-min-n' threats surface the caveats, and the current sample has no signal-kill rows, so impact is cosmetic; but the strata-pooling rule should be honored or documented as a deliberate aggregate.

🟡 LOW endStatePassed evidence default mismatches base outcome — scripts/tb-repair-milestone1.ts

Line 468 sets base.endStatePassed = noFix[0]?.passed ?? null (null when unknown), but line 483 sets evidence.endStatePassed: noFix[0]?.passed ?? true (assumes passed when unknown). If a noFix rollout threw (practically unreachable since CONTROL_ROLLOUTS=3 and the loop runs synchronously), the evidence would claim the end state passed, and admitRow would reject with 'end-state-already-passes' — the wrong rejection reason. The evidence [line 483](https://github.com/tangle-network/agent-eval/blob/f1f03f2da32103d3619fb3f3b26b4942d

🟡 LOW Two digest schemes describe 'the control that screened this row', incomparable across admission paths — src/trace-repair/admission-contract.ts

The contract path stamps AdmittedRow.policyDigest with defineControlPolicy's digest (contentHash over id/stepBudget/scaffold/model/commandTimeoutSeconds), while the executed path uses continuationPolicyDigest (full continuation policy + scaffold renderings). grade.ts:405 assertPolicySymmetry compares a continuation runner's policyDigest against AdmittedRow.policyDigest, so a campaign that runs admission via runAdmission but grades with a runner reporting defineControlPolicy digests (as milestone1 does) would throw RepairArmSymmetryError, and artifacts from the two paths cannot be compared. Each path is internally consistent; this is a consumer footgun. Fix: single digest source of truth or explicit per-path naming in the artifacts.

🟡 LOW assertCriteria does not validate controlScreening — src/trace-repair/admission-contract.ts

assertCriteria validates maxPrefixDivergenceRatio and controlRollouts but not controlScreening. The validation for controlScreening occurs inside assertControlCalibrated (control-policy.ts:144), which is called separately from admitRow at line 198. Not a correctness bug — the validation still fires — but the criteria validation is spread across two functions instead of one, making it easier for a future change to forget one.

🟡 LOW assertCriteria does not validate controlScreening field — src/trace-repair/admission-contract.ts

assertCriteria validates maxPrefixDivergenceRatio and controlRollouts but skips the new controlScreening field on AdmissionCriteria. The screening mode is instead validated inside assertControlCalibrated (called at line 198) via CONTROL_SCREENING_MODES.includes. This works but is fragile: if someone calls assertCriteria standalone (it's not exported, but the pattern is inconsistent), controlScreening would be unchecked. Fix: add controlScreening validation to assertCriteria for completeness, or document that assertControlCalibrated is the sole validator.

🟡 LOW Oracle certification is applied by taskName only, never checked against the graded image or suiteDigest — src/trace-repair/admission.ts

input.taskOracles.get(row.taskName) uses the task name as the sole key, but OracleDeterminismVerdict carries image and suiteDigest (provenance-only per its docstring). A campaign whose local suite bytes or image differ from the certified ones still applies the certification as if it were measured on the bytes being graded; tb-repair-milestone1 also never compares certification.suiteDigest to task.suiteDigest (scripts/tb-repair-milestone1.ts:602). A stale certification can then admit rows whose actual grader is nondeterministic. Fix: key the registry by taskName but cross-check certification.image and certification.suiteDigest against the row/evidence, refusing (or excluding) on mismatch.

🟡 LOW declared-inert screening mode is unreachable through runAdmission — src/trace-repair/admission.ts

runAdmission calls assertControlCalibrated with stepBudget from PinnedContinuationPolicy, which definePinnedContinuationPolicy (continuation-policy.ts:98) requires to be a positive integer (>= 1). controlCanRescue(stepBudget) returns true for any stepBudget > 0. So assertControlCalibrated with screening='declared-inert' always throws UncalibratedControlError. This means: (1) the controlScreening:'declared-inert' option in AdmissionConfigInput is a configuration that always throws when used with the standard pipeline; (2) the control-cannot-rescue threat in delta-repair.ts:176-183 can never fire; (3) the control-passed-on-identical-state rejection in admission-contract.ts:283-289 can never be produced by runAdmission. The pure contract (admitRow in admission-contract.ts) supports this mode

🟡 LOW Empty assertions array not enforced — false stable verdict possible — src/trace-repair/oracle-determinism.ts

The contract (line 64-66) says assertions must be null when the suite reported nothing, and 'Never an empty array standing in for the suite said nothing.' But stateVerdict only checks replicates.every(r => r.assertions !== null) — an empty array passes this check (it is not null). If all replicates carry assertions:[], then perAssertion=true, outcomes map is empty, flipped=[], flipRate=0, granularity='per-assertion', stable=true. The verdict reports 'stable, counted per-assertion' when zero assertions were actually measured. This violates the repo's fail-loud doctrine. Fix: add r.assertions.length > 0 to the perAssertion guard, or explicitly rejec

🟡 LOW oracleDeterminism accepts a single-state certification and missing taskName — src/trace-repair/oracle-determinism.ts

The reducer only requires groups.length>0 and >=2 replicates per group; evidence with groups for only 'unsolved' or only 'solved' is accepted and can certify stable, though the module documents 'the two states certification produces'. parseTaskOracleRegistry/taskOracleRegistry do not validate that a measurement carries a taskName, so a malformed entry becomes a registry key 'undefined' silently. A certifier bug that only measured one state would produce a stable certification instead of failing. Fix: require both states (or a declared opt-out) and reject measurements missing taskName.

🟡 LOW parseTaskOracleRegistry casts measurements array without per-element validation — src/trace-repair/oracle-determinism.ts

Line 303: measurements.map((evidence) => oracleDeterminism(evidence)) casts each element as OracleDeterminismEvidence without checking that it is an object. A non-object in the measurements array (from a malformed JSON file) will reach oracleDeterminism and throw Cannot read properties of undefined instead of a ValidationError naming the problem. Add a loop guard: if (typeof evidence !== 'object' || evidence === null) throw new ValidationError(...).

🟡 LOW declared-inert fixture carries a rescue-capable policy digest — tests/trace-repair/delta-repair.test.ts

inert() spreads measuredRowResult (which sets controlPolicyDigest: POLICY_DIGEST, the digest of CONTROL_POLICY with stepBudget 20) and only overrides controlScreening: 'declared-inert'. That pairing is exactly what assertControlCalibrated refuses (control-policy.ts:158-163: declared-inert + canRescue throws UncalibratedControlError), and in production gradeRepairRow derives both fields from one admitted row that admission already calibrated. The threat under test claims rows were 'screened under a control that makes no model call' (delta-repair.ts:180), which the row's own digest contradicts. The test passes only because collectThreats keys on controlScreening alone, so it does not prove the threat fires on a production-shaped inert row. Fix: set controlPolicyDigest to th

🟡 LOW Oracle fixture measuredAt hardcoded to a near-current date — tests/trace-repair/fixtures.ts

Both stableOracle and flakyOracle hardcode measuredAt: '2026-08-10T00:00:00.000Z'. The field is provenance-only (never compared), so this is functionally correct. However the date matches today's env date, meaning it will appear increasingly stale in future test runs without causing any failure. Consider deriving a fixed epoch or documenting that the value is intentionally pinned. No action required for merge.

🟡 LOW flakyOracle fixture contradicts its own docstring — tests/trace-repair/fixtures.ts

Docstring (lines 84-88) says 'the conjunction fails every time, one term does not', but replicate index 0 has both test_correct and test_speedup[6] passing while reward: '0' and passed: false. A suite whose reward is the conjunction of its reported assertions would write reward '1' on that replicate; this fixture models a grader whose own assertion report contradicts its reward/passed. Harmless to the tests (the module treats passed/reward/assertions as independent raw fields, and only test_speedup's flip drives the 'task-oracle-nondeterministic' rejection at oracle-determinism.test.ts:177), but the fixture does not faithfully model the documented sha

🟡 LOW rewardsObserved fixture replicate is internally contradictory — tests/trace-repair/oracle-determinism.test.ts

Replicate index 1 carries assertions: PASS (both assertions pass) yet passed: false and reward: null. Because both replicates report assertions, stateVerdict takes the per-assertion branch and finds zero assertion flips → stable: true, while passes would be 1/2 — i.e. this exact fixture would certify stable a grader whose reward moved between '1' and missing. The test only asserts rewardsObserved so it passes, but the fixture encodes a grader that contradicts itself, and the latent stable-true/1-of-2-pass mismatch is exactly the masking case the module's reward-vs-assertion distinction exists to expose. Give the second replicate a genuinely failing assertion (or a null assertion report with a '0' reward) so the fixture is internally coherent.

_{tangletools · 2026-08-10T13:48:51Z · trace}

tangletools

❌ 1 Blocking Finding — `f1f03f2d`

Full multi-shot audit completed 8/8 planned shots over 30 changed files. Global verifier still owns final merge decision. | Full multi-shot audit completed 8/8 planned shots over 30 changed files. Global verifier still owns final merge decision. | Full multi-shot audit completed 8/8 planned shots over 30 changed files. Global verifier still owns final merge decision.

Full immutable report for this review: trace

Summary comment for this run: full summary

_{tangletools · 2026-08-10T13:48:51Z · immutable trace}

drewstone added 6 commits August 10, 2026 02:51

fix(trace-repair): refuse a contended replicate count that cannot sho…

a80dd1d

…w a flip A load group of one measures nothing, and the substrate rule already rejects it. Refusing at the flag names the mistake instead of failing three phases in.

feat(trace-repair): build stop points without the measurement seat

33bf3f9

Reconstructing a recorded stop point needs containers and no model call. A campaign queued behind the seat builds them first, so the seat's time is spent on model calls alone.

tangletools previously approved these changes Aug 10, 2026

View reviewed changes

tangletools reviewed Aug 10, 2026

View reviewed changes

drewstone dismissed tangletools’s stale review via f1f03f2 August 10, 2026 10:31

tangletools approved these changes Aug 10, 2026

View reviewed changes

tangletools reviewed Aug 10, 2026

View reviewed changes

tangletools approved these changes Aug 10, 2026

View reviewed changes

tangletools reviewed Aug 10, 2026

View reviewed changes

tangletools approved these changes Aug 10, 2026

View reviewed changes

tangletools reviewed Aug 10, 2026

View reviewed changes

tangletools requested changes Aug 10, 2026

View reviewed changes

Uh oh!

Conversation

drewstone commented Aug 10, 2026

What runs

Design points that are load-bearing

Replay fidelity, measured

Pre-registration

Proof

Uh oh!

tangletools left a comment

Choose a reason for hiding this comment

✅ Auto-approved drewstone PR — 33bf3f91

Uh oh!

tangletools left a comment

Choose a reason for hiding this comment

🟡 Value Audit — sound-with-nits

💰 Value — sound

🎯 Usefulness — sound-with-nits

🔎 Heuristic Signals

🎯 Usefulness Audit

💰 Value Audit

Uh oh!

drewstone commented Aug 10, 2026

Measured: unconditional continuation rescues 3 of 64 rollouts (4.7 %), 2 of 32 rows (6.2 %)

Uh oh!

tangletools commented Aug 10, 2026

⚠️ Review Interrupted — 33bf3f91

Uh oh!

tangletools left a comment

Choose a reason for hiding this comment

✅ Auto-approved drewstone PR — f1f03f2d

Uh oh!

tangletools left a comment

Choose a reason for hiding this comment

🟢 Value Audit — sound

💰 Value — sound

🎯 Usefulness — sound

🔎 Heuristic Signals

💰 Value Audit

Uh oh!

tangletools commented Aug 10, 2026

⚠️ Review Incomplete — f1f03f2d

Uh oh!

tangletools left a comment

Choose a reason for hiding this comment

✅ Auto-approved drewstone PR — f1f03f2d

Uh oh!

tangletools left a comment

Choose a reason for hiding this comment

🟡 Value Audit — sound-with-nits

💰 Value — sound-with-nits

🎯 Usefulness — sound

🔎 Heuristic Signals

🎯 Usefulness Audit

💰 Value Audit

Uh oh!

tangletools commented Aug 10, 2026

⚠️ Review Incomplete — f1f03f2d

Uh oh!

tangletools left a comment

Choose a reason for hiding this comment

✅ Auto-approved drewstone PR — f1f03f2d

Uh oh!

tangletools left a comment

Choose a reason for hiding this comment

🟠 Value Audit — better-approach-exists

💰 Value — better-approach-exists

🎯 Usefulness — sound

🔎 Heuristic Signals

💰 Value Audit

Uh oh!

tangletools commented Aug 10, 2026

❌ Needs Work — f1f03f2d

Blocking

Other

Uh oh!

tangletools left a comment

Choose a reason for hiding this comment

❌ 1 Blocking Finding — f1f03f2d

Uh oh!

✅ Auto-approved drewstone PR — `33bf3f91`

⚠️ Review Interrupted — `33bf3f91`

✅ Auto-approved drewstone PR — `f1f03f2d`

⚠️ Review Incomplete — `f1f03f2d`

✅ Auto-approved drewstone PR — `f1f03f2d`

⚠️ Review Incomplete — `f1f03f2d`

✅ Auto-approved drewstone PR — `f1f03f2d`

❌ Needs Work — `f1f03f2d`

❌ 1 Blocking Finding — `f1f03f2d`