Skip to content

Fix balanced few-shot sampling: falsy label truncation and global RNG use#1310

Open
ErenAta16 wants to merge 1 commit into
huggingface:mainfrom
ErenAta16:fix/balanced-fewshot-falsy-label
Open

Fix balanced few-shot sampling: falsy label truncation and global RNG use#1310
ErenAta16 wants to merge 1 commit into
huggingface:mainfrom
ErenAta16:fix/balanced-fewshot-falsy-label

Conversation

@ErenAta16

Copy link
Copy Markdown

Closes #1309.

What

FewShotSampler._init_fewshot_sampling_balanced (src/lighteval/tasks/prompt_manager.py) had two issues in the same function.

1. A falsy class label silently truncates the balanced sample. The label-cycle guard was if not next_label: break. labels_iterable is a cycle, which never yields None for a non-empty pool, so the guard was meant for the empty-pool case, but if not next_label also fires on a valid but falsy label (integer 0, empty string, False). Labels come from instance.fewshot_sorting_class or as_list(instance.get_golds())[0], so any task with a falsy label had its balanced selection cut short, down to zero examples if that label sorted first.

2. Balanced sampling used the global random module. Unlike the sibling _init_fewshot_sampling_random (local random.Random(variance_seed)), the balanced path called random.seed/random.shuffle/random.randrange on the global module, which is non-reproducible if other code touches the global RNG and leaks state into the rest of the program.

Reproduction (before)

from lighteval.tasks.prompt_manager import FewShotSampler
from lighteval.tasks.requests import Doc

class FakeTask:
    fewshot_selection = "balanced"; fewshot_split = "test"
    def __init__(self, docs): self._docs = docs
    def fewshot_docs(self): return self._docs

docs  = [Doc(query=f"q{i}", choices=["", "x"], gold_index=0) for i in range(20)]  # label ""
docs += [Doc(query=f"p{i}", choices=["", "x"], gold_index=1) for i in range(20)]  # label "x"
print(len(FewShotSampler(FakeTask(docs)).sample_fewshot_examples(10, variance_seed=0)))  # 0, expected 10

Fix

  1. if not next_label -> if next_label is None (a cycle only yields None when sorted_labels is empty; falsy-but-valid labels no longer truncate).
  2. Use a local rnd = random.Random(variance_seed) and its .shuffle/.randrange, mirroring _init_fewshot_sampling_random.

Tests

Added two regression tests to tests/unit/prompt/test_prompt_manager.py:

  • test_balanced_fewshot_falsy_label_not_truncated: a task with an empty-string-gold class returns the full num_fewshot with both classes represented.
  • test_balanced_fewshot_does_not_disturb_global_rng: a balanced sampling call leaves the global random stream untouched.

RED/GREEN verified: both fail on main and pass with the fix. test_prompt_manager.py is green (5 passed), ruff check clean on both files.

… use

_init_fewshot_sampling_balanced had two issues in the same function:

1. The label-cycle guard "if not next_label" broke not just when the pool
   was empty but also on a valid falsy label (integer 0, empty string,
   False). Labels come from fewshot_sorting_class or the gold, so any task
   with such a label had its balanced sample cut short, down to zero if the
   falsy label sorted first. Use "if next_label is None" (a cycle only
   yields None when sorted_labels is empty).

2. It seeded and drew from the global random module, unlike the sibling
   _init_fewshot_sampling_random which uses a local Random. That is
   non-reproducible if other code touches the global RNG and leaks state.
   Use a local random.Random(variance_seed).

Adds regression tests for both: a falsy-label task now gets the full
num_fewshot with both classes represented, and a balanced call leaves the
global RNG stream untouched.

Closes huggingface#1309
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Balanced few-shot sampling truncates on a falsy class label and uses the global random module

1 participant