Problem
The generic DSPy RLM path can silently lose valid findings when the model calls SUBMIT(answer, findings) with a Python list.
DSPy declares findings_json as a str output.
Its final-output parser converts the list with str(value).
The RPC bridge then receives Python repr with single quotes.
_parse_findings_json rejects that repr as JSON and returns an empty findings array.
The repair turn only sees the prose answer, so it can also return [].
Observed environment
@tangle-network/agent-eval 0.144.6agent-eval-rpc[dspy] 0.144.6- DSPy 3.2.1
- GLM 5.2 through an OpenAI-compatible CLI Bridge
The model inspected the correct spans and built three valid finding objects.
Its final code was SUBMIT(answer, findings).
The engine returned zero findings and requireStructuredFindings failed.
Expected
A list-valued findings_json submission must retain all valid rows or fail with a precise type error.
It must not silently become an empty findings array.
Suggested fix
Use a typed findings list in the DSPy signature, or serialize list and mapping values with canonical JSON before _parse_findings_json.
Also tell the RLM action prompt to call SUBMIT(answer, json.dumps(findings)) when the output field remains a string.
Add a regression test where SUBMIT receives a Python list containing one valid cited finding.
Problem
The generic DSPy RLM path can silently lose valid findings when the model calls
SUBMIT(answer, findings)with a Python list.DSPy declares
findings_jsonas astroutput.Its final-output parser converts the list with
str(value).The RPC bridge then receives Python repr with single quotes.
_parse_findings_jsonrejects that repr as JSON and returns an empty findings array.The repair turn only sees the prose answer, so it can also return
[].Observed environment
@tangle-network/agent-eval0.144.6agent-eval-rpc[dspy]0.144.6The model inspected the correct spans and built three valid finding objects.
Its final code was
SUBMIT(answer, findings).The engine returned zero findings and
requireStructuredFindingsfailed.Expected
A list-valued
findings_jsonsubmission must retain all valid rows or fail with a precise type error.It must not silently become an empty findings array.
Suggested fix
Use a typed findings list in the DSPy signature, or serialize list and mapping values with canonical JSON before
_parse_findings_json.Also tell the RLM action prompt to call
SUBMIT(answer, json.dumps(findings))when the output field remains a string.Add a regression test where
SUBMITreceives a Python list containing one valid cited finding.