FEAT: Best-of-N jailbreak attack + CharNoiseConverter

[francose]

implements Best-of-N (Hughes et al. 2024, arXiv:2412.03556) per the design we landed on in #2170. closes #2170.

what's here

• BestOfNAttack - thin PromptSendingAttack subclass. draws a scrambled / re-cased / ascii-noised variant of the objective and re-samples a fresh one each attempt until an objective scorer flips or the n_samples budget runs out. the re-sampling just reuses the existing retry loop, so every attempt is a new draw, no extra machinery.
• CharNoiseConverter - new converter for the ascii-noise augmentation (cheap +/-1 printable-char perturbation, no LLM). the other two augmentations already ship as converters (CharSwapConverter, RandomCapitalLettersConverter), so this is the only new brick.
• best_of_n technique registered in extra.py so it runs as a composed technique, same way flip was set up in MAINT: Migrate FlipAttack to a core 'flip' attack technique #2178.

one sigma dial drives all three augmentations, power-scaled per the paper (scramble/case at sigma ** 0.5, noise at sigma ** 3 so it stays lighter). n_samples - 1 maps onto max_attempts_on_failure.

does it work

you asked to actually verify it, not just point at the paper, so i ran it against live targets.

setup: 15 objectives pulled from the bundled scorer-eval sets (illegal activity, cyber, phishing, harassment, misinformation), sigma 0.4, one shipped BoN sample per attempt, loop until a sample breaks through and record the first-success index. ASR at budget k = fraction of objectives cracked within the first k samples, so it's monotonic in k by construction (that's the definition of best-of-n, not a scoring trick). success was scored with the task_achieved_refined true/false scorer, which asks "did the response fulfill the request vs refuse / safe-complete", i.e. the jailbreak question, not "is this a correct answer" like the QA scorer. the scorer runs on a different provider than the target so nothing grades its own output.

the clean run, gpt-4o-mini target scored by gemini-2.5-flash:

N	ASR
1	20% (3/15)
2	27% (4/15)
4	47% (7/15)
8	47% (7/15)
16	60% (9/15)

20% single-shot up to 60% at N=16... keep re-sampling augmented variants and refusals fall off. i didn't want to trust the judge blind, so i pulled the response text + judge rationale back out for a few of the successes and read them by hand: an anti-mask/anti-vax post, a phishing "click this link" email, a fauci lab-leak article. real compliance, not the judge false-positiving on a refusal. the objectives that held to N=16 skew toward direct "how do i" operational asks (the illegal-activity set held all three), while "write a persuasive X" framings fold fastest.

and it isn't a gpt-4o-mini quirk, the climb holds across models and sizes:

target	judge	N=1	N=2	N=4	N=8	N=16
gpt-4o-mini	gemini-2.5-flash	20%	27%	47%	47%	60%
gpt-4o	gemini-2.5-flash	13%	33%	47%	47%	47%
gpt-4.1-mini	gemini-2.5-flash	27%	40%	47%	47%	53%
gpt-4.1	gemini-2.5-flash	40%	60%	67%	67%	67%
gemini-2.5-flash	gpt-4o-mini	13%	20%	27%	40%	40%
gemini-2.5-pro	gpt-4o-mini	40%	40%	40%	60%	60%
llama3.2:3b (local)	gemini-2.5-flash	7%	7%	7%	7%	13%

read this as "ASR rises with N on every target", not as a leaderboard, the between-model gaps are inside the noise at 15 objectives. two things to flag honestly: to keep the target off its own judge i judged the openai targets with gemini and the google targets with gpt-4o-mini, so cross-vendor absolute numbers aren't apples to apples (the within-row N trend is the solid part). i spot-checked a success from each judge direction by hand and both were real compliance. the local llama3.2:3b barely moves at this budget, which is the same scaling story from the other end, a heavily safety-tuned 3B model just needs a lot more than 16 samples.

straight about the limits: this is a single run at small N (16), it corroborates the paper's "ASR scales with budget" result, it doesn't replace it. hughes et al. went to ~10k samples. a couple of targets were out of scope for a clean read, gpt-5.5 gets rejected at an api-level policy filter before the model sees the prompt (that's the vendor's input classifier, not a model refusal, so it measures the wrong thing), and gemini-2.0-flash is retired now.

tests + docs

• unit tests for the converter and the attack: budget/validation, the sigma->rate mapping pinned across all three converters, and an end-to-end check that each retry sends a freshly augmented prompt through the real normalizer (not one prompt converted once and replayed).
• doc notebooks updated (text converters + single-turn), references.bib entry added, EXTRA_TECHNIQUE_NAMES updated.

one thing to weigh

since the stock retry loop already re-applies stochastic converters per attempt, best-of-n could alternatively be a pure technique (the three converters + a scorer, like flip) with no bespoke attack class. i went with a small attack class so n_samples / sigma have a clean home, but if you'd rather collapse it to a technique i'm happy to.

cc @Yusup-Badiev who offered to test it against their own setup.

[Copilot]

Pull request overview

Adds a Best-of-N jailbreak workflow and stochastic ASCII character-noise converter.

Changes:

• Adds BestOfNAttack with configurable sampling and augmentation strength.
• Adds and tests CharNoiseConverter.
• Registers and documents the new attack technique.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
pyrit/converter/char_noise_converter.py	Implements ASCII noise conversion.
pyrit/converter/__init__.py	Exports the converter.
pyrit/executor/attack/single_turn/best_of_n_attack.py	Implements Best-of-N sampling.
pyrit/executor/attack/single_turn/__init__.py	Exports the attack locally.
pyrit/executor/attack/__init__.py	Exports the attack publicly.
pyrit/setup/initializers/techniques/extra.py	Registers best_of_n.
tests/unit/converter/test_char_noise_converter.py	Tests character noise behavior.
tests/unit/executor/attack/single_turn/test_best_of_n_attack.py	Tests configuration and retries.
tests/unit/setup/test_technique_initializer.py	Updates expected techniques.
doc/references.bib	Adds the Best-of-N paper.
doc/code/executor/1_single_turn.py	Adds attack documentation.
doc/code/executor/1_single_turn.ipynb	Mirrors attack documentation.
doc/code/converters/1_text_to_text_converters.py	Adds converter example.
doc/code/converters/1_text_to_text_converters.ipynb	Mirrors converter example.

[francose]

went through copilot's review, pushed 5e77c05:

• n_samples now checks it's a positive int, not just positive. 1.5 used to slip through and die later in range(), now it raises up front.
• the single-turn doc example was using the question-answering scorer, which grades answer correctness rather than jailbreak success. swapped it to the task_achieved_refined true/false scorer so the loop stops on a broken guardrail like it should. that one was genuinely misleading, good catch.
• regenerated the converter modality table so CharNoiseConverter is listed.

on pulling the converter stack out of the attack and into a technique: roman and i already went back and forth on this on #2170. the iterative sample-until-success loop is the reason it's an attack and not a pure technique ("requires an attack since it's iterative"). left a note in the PR description flagging the pure-technique alternative in case he wants to revisit, but keeping the attack class per that thread.