Continuing the safety classifier set after LlamaGuard (#1867) and ShieldGemma (#2261), this proposes a WildGuard scorer (Allen Institute, NeurIPS 2024). WildGuard is a useful third because it is a single model that judges a prompt and response together, which is a different shape from the other two, so it exercises the composition API in a new way. Opening this to settle the design and one hosting question before writing code.
How WildGuard differs from the first two
WildGuard takes a prompt and a response together and returns three labels in one call:
Harmful request: yes/no
Response refusal: yes/no
Harmful response: yes/no
Two consequences for a PyRIT scorer:
- It emits three judgements, but a
TrueFalseScorer returns one boolean. Since all three come from a single model call, running three separate scorers would repeat the same request three times.
- It needs the originating prompt as well as the response being scored. The other two classifiers judge a single message, so this is the first that needs the pair.
Proposed shape, still mirroring the existing scorers
pyrit/score/true_false/wildguard_parser.py: parse_wildguard_response, reading the three labelled lines into a dict, raising InvalidJsonException if the expected labels are absent so the retry applies.
pyrit/score/true_false/wildguard_scorer.py: WildGuardScorer(TrueFalseScorer) composing CallableResponseHandler, plus a WildGuardLabel enum selecting which of the three judgements is the scored value.
pyrit/datasets/score/wildguard/: the request template.
- Tests for the parser and scorer.
The value returned is the selected label, with all three parsed labels kept on score_metadata so a single call surfaces the full result. Default WildGuardLabel.HARMFUL_RESPONSE, since scoring a model response for harm is the common case and matches the response-side default we chose for ShieldGemma.
Open questions
- Sourcing the prompt. WildGuard needs the request that produced the response being scored. The cleanest options I see are to read it from
objective, or to accept it as an explicit scoring parameter. Reading conversation history from memory is possible but heavier. Which fits PyRIT's scoring model best?
- One scorer with a label selector, versus three thin scorers. Given all three labels come from one call, I lean toward a single scorer with a
WildGuardLabel selector and the other two labels in metadata, so callers do not pay for three requests to read three fields. Does that match your preference?
- Validation hosting. This is the one place WildGuard is harder than ShieldGemma. It is gated on HuggingFace, has no serverless inference provider, and is not in the Ollama library, so the local path that worked for ShieldGemma does not apply directly. The model is a 7B Mistral fine tune, so it can run locally once the weights are obtained, but that needs license acceptance and a quantization step. Do you have a preferred way to validate this one, or would unit tests plus a documented local serving recipe be acceptable for the first pass, with a live transcript to follow?
Happy to start once the prompt sourcing question in particular is settled.
Continuing the safety classifier set after LlamaGuard (#1867) and ShieldGemma (#2261), this proposes a WildGuard scorer (Allen Institute, NeurIPS 2024). WildGuard is a useful third because it is a single model that judges a prompt and response together, which is a different shape from the other two, so it exercises the composition API in a new way. Opening this to settle the design and one hosting question before writing code.
How WildGuard differs from the first two
WildGuard takes a prompt and a response together and returns three labels in one call:
Two consequences for a PyRIT scorer:
TrueFalseScorerreturns one boolean. Since all three come from a single model call, running three separate scorers would repeat the same request three times.Proposed shape, still mirroring the existing scorers
pyrit/score/true_false/wildguard_parser.py:parse_wildguard_response, reading the three labelled lines into a dict, raisingInvalidJsonExceptionif the expected labels are absent so the retry applies.pyrit/score/true_false/wildguard_scorer.py:WildGuardScorer(TrueFalseScorer)composingCallableResponseHandler, plus aWildGuardLabelenum selecting which of the three judgements is the scored value.pyrit/datasets/score/wildguard/: the request template.The value returned is the selected label, with all three parsed labels kept on
score_metadataso a single call surfaces the full result. DefaultWildGuardLabel.HARMFUL_RESPONSE, since scoring a model response for harm is the common case and matches the response-side default we chose for ShieldGemma.Open questions
objective, or to accept it as an explicit scoring parameter. Reading conversation history from memory is possible but heavier. Which fits PyRIT's scoring model best?WildGuardLabelselector and the other two labels in metadata, so callers do not pay for three requests to read three fields. Does that match your preference?Happy to start once the prompt sourcing question in particular is settled.