`anthropic_chat()` crashes with IndexError when API response content is empty or filtered

[thehalleyyoung]

anthropic_chat() crashes with IndexError when API response content is empty or filtered

Repository: microsoft/StableQAT

Description

The anthropic_chat() function in lm_eval/models/anthropic_llms.py accesses response.content[0].text without verifying that response.content is non-empty or that the first content block is a TextBlock:

def messages():
    response = client.messages.create(
        model=model,
        max_tokens=max_tokens,
        temperature=temperature,
        messages=[{"role": "user", "content": f"{prompt}"}],
        **kwargs,
    )
    return response.content[0].text  # line 139

This crashes with:

• IndexError if response.content is an empty list (e.g., the model's response was filtered by Anthropic's safety systems, or the API returned a stop_reason of "end_turn" with no content)
• AttributeError if content[0] is not a TextBlock (e.g., it's a ToolUseBlock in newer API versions)

Content filtering is increasingly common with safety-tuned Anthropic models, making this a realistic failure mode during evaluation.

Suggested fix

def messages():
    response = client.messages.create(
        model=model,
        max_tokens=max_tokens,
        temperature=temperature,
        messages=[{"role": "user", "content": f"{prompt}"}],
        **kwargs,
    )
    if not response.content:
        return ""
    block = response.content[0]
    return block.text if hasattr(block, "text") else ""

How this was found

This was identified via static analysis using a3-python, which flagged the unguarded index access as a DSE-confirmed reachable NULL_PTR (potential IndexError/AttributeError).