[OlympiadBench] Add separate official English text-only task - #2
Open
taivu1998 wants to merge 4 commits into
Open
[OlympiadBench] Add separate official English text-only task#2taivu1998 wants to merge 4 commits into
taivu1998 wants to merge 4 commits into
Conversation
taivu1998
marked this pull request as ready for review
July 6, 2026 00:34
Use the pinned English open-ended text-only subsets and official scoring and prompt semantics.\n\nFix multi-answer handling, scorer attribution, lean-extra coverage, and result metadata.
Use the benchmark authors' answer-type wording, multiple-answer format, and unit placement semantics.\n\nCover the rendered single-, composite-, and unit-bearing prompts through the generation boundary.
Preserve the existing OlympiadBench task and expose the full English open-ended text-only protocol as OlympiadBench_official.\n\nGive the variant its own config, extra, lean-import coverage, protocol metadata, scorer provenance, and behavioral tests.
taivu1998
force-pushed
the
tdv/olympiadbench
branch
from
July 19, 2026 02:13
88fa33b to
748c538
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add
OlympiadBench_officialalongside the existingOlympiadBenchtask, whose implementation and scoring behavior remain unchanged. This is deliberately a separate benchmark identity rather than a bug-fix rewrite: changing the dataset population, prompt, and grader in place would silently make newOlympiadBenchscores incomparable with existing results.Compared with
OlympiadBenchonmain, which defaults to a bundled 30-problem sample, a generic boxed-answer prompt, and correct-if-anyhendrycks_mathgrading (current adapter, current grader),OlympiadBench_official:OE_TO_maths_en_COMPand 236 physics rows fromOE_TO_physics_en_COMP. The authors link this updated dataset release and defineOE,TO, andenas open-ended, text-only, and English (dataset release, subset taxonomy);The separate task name preserves historical compatibility while giving users an explicit, reproducible path for the authors' English text-only protocol. It includes its own single-task config and install extra, and both OlympiadBench variants remain importable in the torch-free lean installation.