Skip to content

Latest commit

 

History

1 Commit

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 

Repository files navigation

log-report (fixed Terminal-Bench 2 / Harbor task)

Fixed version of the broken dynamo/log-report task.

What was broken

  • Format - task.toml set artifacts = "/app/out.json": a string where Harbor's schema wants a list, pointing at a file the task never produces.
  • Environment - the Dockerfile used the floating tag python:latest, and it copied solution_hint.py (a full reference solution) into the agent image.
  • Verifier - only checked that report.json existed and was non-empty, so echo '{}' scored 1.0. It also wrote the reward to /app/reward.txt instead of /logs/verifier/reward.txt, and never emitted ctrf.json.
  • Instruction - never named the output path, the filename, the JSON keys, or any success criteria.

What was fixed

  • artifacts = ["/app/report.json"], matching what the task actually writes.
  • Base image pinned by digest, solution_hint.py deleted from the build context.
  • Verifier parses the JSON and asserts the real values (6, 3, /index.html), writes reward to /logs/verifier/reward.txt and CTRF to /logs/verifier/ctrf.json.
  • instruction.md rewritten with four numbered success criteria, mapped one-to-one onto four tests.

Calibration

harbor run -p log-report --agent oracle   # reward 1.0
harbor run -p log-report --agent nop      # reward 0.0

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages