Skip to content

fix: correct regex group extraction in browsecomp_eval.py - #93

Open
Neph0s wants to merge 1 commit into
openai:mainfrom
Neph0s:main
Open

fix: correct regex group extraction in browsecomp_eval.py#93
Neph0s wants to merge 1 commit into
openai:mainfrom
Neph0s:main

Conversation

@Neph0s

@Neph0s Neph0s commented Jul 4, 2025

Copy link
Copy Markdown

Problem

The grade_sample method in browsecomp_eval.py has a regex bug that prevents correct grading
evaluation.

Current code:

match = re.search(r"correct: (yes|no)", grading_response)
return match.group(0) if match else "no"  # Returns "correct: yes" or "correct: no"

...
grade_result = self.grade_sample(problem, answer, response_text)

# Metrics based on grading response
is_correct = grade_result == "yes"
is_incorrect = grade_result == "no"

Issue:
- match.group(0) returns the entire match ("correct: yes" or "correct: no")
- But the code later compares grade_result == "yes" which always fails
- This causes is_correct and is_incorrect metrics to always be False

Solution

Change match.group(0) to match.group(1) to extract the captured group:

match = re.search(r"correct: (yes|no)", grading_response)
return match.group(1) if match else "no"  # Returns "yes" or "no"

Impact

- Fixes broken grading logic that was always marking responses as incorrect

Testing

The fix ensures that:
- Input "The answer is correct: yes"returns "yes" (was "correct: yes")
- Input "The answer is correct: no"returns "no" (was "correct: no")
- Comparisons grade_result == "yes" now work correctly

JoyboyBrian added a commit to Osmosis-AI/simple-evals that referenced this pull request Jul 14, 2026
grade_sample returned match.group(0) ("correct: yes"/"correct: no"),
which never equals the bare "yes"/"no" the aggregation compares
against, so is_correct and is_incorrect were always False and reported
accuracy was always 0. Return the captured group instead.

Same one-line fix as unmerged upstream PRs openai#67 and
openai#93.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
JoyboyBrian added a commit to Osmosis-AI/simple-evals that referenced this pull request Jul 14, 2026
* Add BrowseComp web-search wrapper for the Responses API

Upstream simple-evals cannot produce meaningful BrowseComp results:

1. grade_sample returns match.group(0) ("correct: yes"), while
   aggregation compares against the bare string "yes", so reported
   accuracy is always 0 (upstream PRs openai#67 / openai#93, still unmerged).
2. No sampler supports web search, so the official entrypoint runs
   this browsing benchmark closed-book; questions were explicitly
   filtered at construction time to be unanswerable without browsing
   (GPT-4o closed-book: 0.6%).

This wrapper adds a Responses API sampler with the hosted web_search
tool, accepts both grader output forms when aggregating, and provides
a deterministic oracle mode to validate the grading pipeline before
paying for search runs. Reference answers are never written in
plaintext to result files.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Fix BrowseComp grader regex group extraction

grade_sample returned match.group(0) ("correct: yes"/"correct: no"),
which never equals the bare "yes"/"no" the aggregation compares
against, so is_correct and is_incorrect were always False and reported
accuracy was always 0. Return the captured group instead.

Same one-line fix as unmerged upstream PRs openai#67 and
openai#93.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant