fix(rubric): enforce structured output for judge via tool_choice - #85
Open
AUTHENSOR wants to merge 1 commit into
Open
fix(rubric): enforce structured output for judge via tool_choice#85AUTHENSOR wants to merge 1 commit into
AUTHENSOR wants to merge 1 commit into
Conversation
The rubric judge parsed raw LLM text via json.loads (rubric.py:199,286) with no structured output enforcement. An agent transcript interpolated bare into the prompt could inject a JSON verdict that passes json.loads and jsonschema.validate. This converts the output_schema into a ToolInfo and uses tool_choice='required' so the provider enforces the schema natively, with a json.loads fallback for providers without tool support.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The rubric judge in
evaluate_rubric(docent_core/docent/ai_tools/rubric/rubric.py) parses the LLM judge output viajson.loads(output.first_text)at two sites (the streaming callback at line 199 and the return path at line 286). The LLM call does not enforce structured output — noresponse_format, notools, notool_choice. The agent transcript (ar.to_text_new()) is interpolated into the judge prompt via.format().This means an agent that embeds a valid JSON verdict matching the output schema in its transcript can cause the judge to output that JSON, passing both
json.loadsandjsonschema.validate.Fix
Three changes (+55/-3 lines):
_schema_to_tool_info: converts the rubricoutput_schemainto aToolInfofor structured-output enforcement._extract_output: extracts the verdict fromtool_calls[0].arguments(preferred) with ajson.loadsfallback for providers that do not support tools.evaluate_rubric: passestools=[judge_tool], tool_choice='required'toget_llm_completions_async, so the provider enforces the schema natively rather than relying on text parsing.Verification
python3 -m py_compile: compiles OKblack --check --line-length=100: cleanjson.loadsfallback in_extract_outputpreserves compatibility for providers without structured output support