dataset: make all 199 features gradeable, and stop wiping prebuilt images

[akhatua2]

The invariant

Every feature must fail on the base commit and pass with its gold patch. Neither half was safe to assume: pallets_jinja/1621 f5 passed 6/6 on an untouched tree, so it scored 5/5 across archived runs regardless of what the agent wrote.

All 199 features were checked by running the real grading path (runner.sh, one sandbox per feature). 24 were broken, and none were findable by reading the specs:

cause	features
runner never invoked the feature's tests	12
dependency drift (litellm, pyarrow, scipy)	8
f1's expectations present in every test patch	5
tests did not discriminate	2
overlapping hunks	1

199/199 now verified.

No gold patch is touched

Where a fix had a choice, it went to the spec or the tests — never the reference. 23 feature.md files are amended where a requirement was genuinely unknowable without reading the tests: unnamed API identifiers (three Go features named no identifier at all), verbatim-compared error messages, and typst's diagnostic source range, which the harness compares as strictly as the message text.

One defect is logged unfixed: pillow/290 f4's reference does not honour its own error_threshold (measured 59.34 at a threshold of 30.0; the old assertions passed only because uint8 arithmetic wrapped). Fixing it would mean editing feature.patch.

Images were recompiling everything, every run

sandbox.py ran git clean -fdx before each graded feature. -x deletes gitignored paths — which is exactly where the images keep the build output they pre-compile at the base commit:

# typst Dockerfile: "Pre-build everything at base commit (cache layer)"
RUN cargo build --package typst --package typst-cli && \
    cargo build --package typst-tests --tests

Result: 335 crate compiles per graded run, against a runner.sh that already said git clean -fd # No -x to preserve target/. The dataset author anticipated this and the harness overrode it. Dropped -x here and in the four runners whose EXIT trap did the same; all four images rebuilt and re-verified (tiktoken OK 10, pillow 25/290/68 OK 5 each).

Measured share of pair wall-clock: 12% for typst, 0% for tiktoken and pillow — a real inefficiency, but not the reason runs are slow. That is ~10 s per agent step.

Also

• 8 published images were arm64-only, so Modal (amd64-only) could not run them; all now multi-arch.
• dottxt/1706 fetched its fixture models live — a failed download ERRORed 10 tests on base and gold, and the runner fails on any ERROR. Models baked in, plus HF_HUB_DISABLE_XET=1 (the repo sets filterwarnings = ["error"] and hf_xet drifted ahead of huggingface_hub, so any download raised a DeprecationWarning as an error).
• Per-feature reasoning for every change is in dataset/SPEC_AUDIT.md.
• scripts/check_gradeable.py reproduces the whole sweep.

[akhatua2]

Scope note, to be precise about what "grading" means here:

• The decision rule is untouched. test_merged — create agent1/agent2 branches, git merge --no-commit --no-ff, test the merged tree against both feature suites — is byte-identical. git diff over src/cooperbench/ is 2 files: the version string and one flag.
• The one harness change is git clean -fdx -> git clean -fd, which only controls whether the image's pre-compiled build output survives into the test run. It changes speed, not pass/fail.
• Everything else is dataset: feature.md, tests.patch, runner.sh, Dockerfile, and rebuilt images.

Measured outcomes do change for the 24 features that were previously ungradeable — that is the intended fix, and the reason 0.0.28 scores should not be compared against 0.0.29.