Skip to content

Fix sequential few-shot sampling mutating the shared few-shot pool#1308

Open
ErenAta16 wants to merge 1 commit into
huggingface:mainfrom
ErenAta16:fix/sequential-fewshot-shared-pool-mutation
Open

Fix sequential few-shot sampling mutating the shared few-shot pool#1308
ErenAta16 wants to merge 1 commit into
huggingface:mainfrom
ErenAta16:fix/sequential-fewshot-shared-pool-mutation

Conversation

@ErenAta16

Copy link
Copy Markdown

Fixes #1307.

FewShotSampler._init_fewshot_sampling_sequential rotated the list returned by task.fewshot_docs() in place to offset the selection per seed. That list is the task's memoized _fewshot_docs, returned by reference, so the rotation mutated shared state. When an evaluation runs several few-shot seeds against the same task (variance estimation), each seed rotated the already-rotated pool, so the offsets accumulated and every seed after the first selected the wrong examples. The per-seed cache stores a reference rather than a copy, so the cached selections all aliased one over-rotated list, and the pool itself was left rotated after the run.

The fix copies the pool before rotating, matching what _init_fewshot_sampling_random already does. Only the sequential method is affected: random copies the pool, and balanced builds its own per-label lists and never mutates it.

Repro on main before the fix:

seed 0: picked=['0', '1']  expected=['0', '1']
seed 1: picked=['2', '3']  expected=['2', '3']
seed 2: picked=['6', '7']  expected=['4', '5']
pool after sampling: ['6', '7', '8', '9', '0', '1', '2', '3', '4', '5']
pool unchanged: False

Tests

The existing test_fewshot_sampler[sequential] compared the result against fewshot_docs() after sampling, so it read back the mutated pool and passed despite the bug. I updated it to check the rotation against the original order, and added two regression tests:

  • test_sequential_fewshot_does_not_mutate_shared_pool: the memoized pool is unchanged after sampling.
  • test_sequential_fewshot_seeds_are_independent: each seed selects the correct rotation from the original order.

Both new tests fail on main and pass with the fix.

pytest tests/unit/prompt/test_prompt_manager.py

`_init_fewshot_sampling_sequential` rotated the list returned by
`fewshot_docs()` in place. That list is the task's memoized
`_fewshot_docs`, returned by reference, so the rotation mutated shared
state: across variance seeds the offsets accumulated (every seed after
the first selected the wrong examples) and the per-seed cache entries all
aliased one over-rotated list. Copy the pool before rotating, matching
`_init_fewshot_sampling_random`.

The existing sequential test compared against `fewshot_docs()` after
sampling, so it read back the mutated pool and passed despite the bug;
update it to check the rotation against the original order, and add
regression tests for pool integrity and seed independence.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Sequential few-shot selection mutates the task's shared few-shot pool in place, corrupting later variance seeds

1 participant