Skip to content

[OlympiadBench] Add separate official English text-only task - #2

Open
taivu1998 wants to merge 4 commits into
marin-community:mainfrom
taivu1998:tdv/olympiadbench
Open

[OlympiadBench] Add separate official English text-only task#2
taivu1998 wants to merge 4 commits into
marin-community:mainfrom
taivu1998:tdv/olympiadbench

Conversation

@taivu1998

@taivu1998 taivu1998 commented Jul 6, 2026

Copy link
Copy Markdown

Add OlympiadBench_official alongside the existing OlympiadBench task, whose implementation and scoring behavior remain unchanged. This is deliberately a separate benchmark identity rather than a bug-fix rewrite: changing the dataset population, prompt, and grader in place would silently make new OlympiadBench scores incomparable with existing results.

Compared with OlympiadBench on main, which defaults to a bundled 30-problem sample, a generic boxed-answer prompt, and correct-if-any hendrycks_math grading (current adapter, current grader), OlympiadBench_official:

  • evaluates exactly 910 English open-ended text-only problems: 674 mathematics rows from OE_TO_maths_en_COMP and 236 physics rows from OE_TO_physics_en_COMP. The authors link this updated dataset release and define OE, TO, and en as open-ended, text-only, and English (dataset release, subset taxonomy);
  • adapts the authors' English open-ended prompt, including answer-type wording, ordered multiple answers, and unit placement (prompt implementation);
  • vendors the authors' automated scorer under its MIT license, preserving complete multi-answer grading and per-problem tolerances for numerical, expression, equation, interval, percentage, and plus/minus equivalence (reference scorer); and
  • pins both dataset and reference-code revisions, rejects scope/schema/row-count drift, records protocol metadata, reports aggregate and Math/Physics metrics, and supports native pass@k.

The separate task name preserves historical compatibility while giving users an explicit, reproducible path for the authors' English text-only protocol. It includes its own single-task config and install extra, and both OlympiadBench variants remain importable in the torch-free lean installation.

@taivu1998
taivu1998 marked this pull request as ready for review July 6, 2026 00:34
Use the pinned English open-ended text-only subsets and official scoring and prompt semantics.\n\nFix multi-answer handling, scorer attribution, lean-extra coverage, and result metadata.
@taivu1998 taivu1998 changed the title Add OlympiadBench open-ended text benchmark adapter [OlympiadBench] Align evaluation with the official English benchmark Jul 19, 2026
@taivu1998 taivu1998 changed the title [OlympiadBench] Align evaluation with the official English benchmark [OlympiadBench] Use full English text-only data and official scoring Jul 19, 2026
Use the benchmark authors' answer-type wording, multiple-answer format, and unit placement semantics.\n\nCover the rendered single-, composite-, and unit-bearing prompts through the generation boundary.
Preserve the existing OlympiadBench task and expose the full English open-ended text-only protocol as OlympiadBench_official.\n\nGive the variant its own config, extra, lean-import coverage, protocol metadata, scorer provenance, and behavioral tests.
@taivu1998 taivu1998 changed the title [OlympiadBench] Use full English text-only data and official scoring [OlympiadBench] Add official-protocol English text-only variant Jul 19, 2026
@taivu1998
taivu1998 force-pushed the tdv/olympiadbench branch from 88fa33b to 748c538 Compare July 19, 2026 02:13
@taivu1998 taivu1998 changed the title [OlympiadBench] Add official-protocol English text-only variant [OlympiadBench] Add separate official English text-only task Jul 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant