Fix balanced few-shot sampling: falsy label truncation and global RNG use#1310
Open
ErenAta16 wants to merge 1 commit into
Open
Fix balanced few-shot sampling: falsy label truncation and global RNG use#1310ErenAta16 wants to merge 1 commit into
ErenAta16 wants to merge 1 commit into
Conversation
… use _init_fewshot_sampling_balanced had two issues in the same function: 1. The label-cycle guard "if not next_label" broke not just when the pool was empty but also on a valid falsy label (integer 0, empty string, False). Labels come from fewshot_sorting_class or the gold, so any task with such a label had its balanced sample cut short, down to zero if the falsy label sorted first. Use "if next_label is None" (a cycle only yields None when sorted_labels is empty). 2. It seeded and drew from the global random module, unlike the sibling _init_fewshot_sampling_random which uses a local Random. That is non-reproducible if other code touches the global RNG and leaks state. Use a local random.Random(variance_seed). Adds regression tests for both: a falsy-label task now gets the full num_fewshot with both classes represented, and a balanced call leaves the global RNG stream untouched. Closes huggingface#1309
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1309.
What
FewShotSampler._init_fewshot_sampling_balanced(src/lighteval/tasks/prompt_manager.py) had two issues in the same function.1. A falsy class label silently truncates the balanced sample. The label-cycle guard was
if not next_label: break.labels_iterableis acycle, which never yieldsNonefor a non-empty pool, so the guard was meant for the empty-pool case, butif not next_labelalso fires on a valid but falsy label (integer0, empty string,False). Labels come frominstance.fewshot_sorting_class or as_list(instance.get_golds())[0], so any task with a falsy label had its balanced selection cut short, down to zero examples if that label sorted first.2. Balanced sampling used the global
randommodule. Unlike the sibling_init_fewshot_sampling_random(localrandom.Random(variance_seed)), the balanced path calledrandom.seed/random.shuffle/random.randrangeon the global module, which is non-reproducible if other code touches the global RNG and leaks state into the rest of the program.Reproduction (before)
Fix
if not next_label->if next_label is None(acycleonly yieldsNonewhensorted_labelsis empty; falsy-but-valid labels no longer truncate).rnd = random.Random(variance_seed)and its.shuffle/.randrange, mirroring_init_fewshot_sampling_random.Tests
Added two regression tests to
tests/unit/prompt/test_prompt_manager.py:test_balanced_fewshot_falsy_label_not_truncated: a task with an empty-string-gold class returns the fullnum_fewshotwith both classes represented.test_balanced_fewshot_does_not_disturb_global_rng: a balanced sampling call leaves the globalrandomstream untouched.RED/GREEN verified: both fail on
mainand pass with the fix.test_prompt_manager.pyis green (5 passed),ruff checkclean on both files.