Skip to content

fix(rubric): enforce structured output for judge via tool_choice - #85

Open
AUTHENSOR wants to merge 1 commit into
TransluceAI:mainfrom
AUTHENSOR:fix/rubric-structured-output
Open

fix(rubric): enforce structured output for judge via tool_choice#85
AUTHENSOR wants to merge 1 commit into
TransluceAI:mainfrom
AUTHENSOR:fix/rubric-structured-output

Conversation

@AUTHENSOR

Copy link
Copy Markdown

Summary

The rubric judge in evaluate_rubric (docent_core/docent/ai_tools/rubric/rubric.py) parses the LLM judge output via json.loads(output.first_text) at two sites (the streaming callback at line 199 and the return path at line 286). The LLM call does not enforce structured output — no response_format, no tools, no tool_choice. The agent transcript (ar.to_text_new()) is interpolated into the judge prompt via .format().

This means an agent that embeds a valid JSON verdict matching the output schema in its transcript can cause the judge to output that JSON, passing both json.loads and jsonschema.validate.

Fix

Three changes (+55/-3 lines):

  1. _schema_to_tool_info: converts the rubric output_schema into a ToolInfo for structured-output enforcement.
  2. _extract_output: extracts the verdict from tool_calls[0].arguments (preferred) with a json.loads fallback for providers that do not support tools.
  3. evaluate_rubric: passes tools=[judge_tool], tool_choice='required' to get_llm_completions_async, so the provider enforces the schema natively rather than relying on text parsing.

Verification

  • python3 -m py_compile: compiles OK
  • black --check --line-length=100: clean
  • Regression PoC (json.loads injection on raw text) now fails after the fix
  • Backward-compat grep: no external callers of changed functions
  • The json.loads fallback in _extract_output preserves compatibility for providers without structured output support

The rubric judge parsed raw LLM text via json.loads (rubric.py:199,286)
with no structured output enforcement. An agent transcript interpolated
bare into the prompt could inject a JSON verdict that passes json.loads
and jsonschema.validate. This converts the output_schema into a ToolInfo
and uses tool_choice='required' so the provider enforces the schema
natively, with a json.loads fallback for providers without tool support.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant