fix: raise eval push payload limit to 100 MiB - #818
Conversation
Computer-use envs (cua_world, osworld_v2) embed a full screenshot per step, so a
single rollout's trace exceeds the 25 MiB per-batch push cap and is dropped
("sample 0 is too large to upload"): observed cua_world ~51 MiB, osworld_v2
~75 MiB at 50 steps, and longer/higher-res rollouts go higher still. #637 raised
this 2 -> 25; raise it to 100 for real headroom on screenshot-heavy rollouts.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
5fdb5dc to
8540da6
Compare
hubert-marek
changed the title
Mirror of PrimeIntellect-ai/prime#818. Computer-use envs (cua_world, osworld_v2) embed a full screenshot per step, so one rollout's sample exceeds the 25 MiB push ceiling and is skipped ("sample 0 is too large to upload"): cua_world ~51 MiB, osworld_v2 ~75 MiB at 50 steps, higher for longer rollouts. Match the Prime Evals client ceiling raised in prime#818. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8540da6ac8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| evaluation_id: str, | ||
| samples: List[Dict[str, Any]], | ||
| max_payload_bytes: int = 25 * 1024 * 1024, | ||
| max_payload_bytes: int = 100 * 1024 * 1024, |
There was a problem hiding this comment.
The commit specifies a 75 MiB upload limit, but both the synchronous and asynchronous defaults are set to 100 MiB. Consequently, batches between 75 and 100 MiB are uploaded instead of being split or skipped at the intended boundary, increasing request size and potentially exceeding the service limit this client-side cap is meant to enforce. Set this default—and the corresponding async default—to 75 * 1024 * 1024.
Useful? React with 👍 / 👎.
1 participant
Follows #637 (which raised the eval-push payload limit 2 → 25 MiB).
Computer-use environments embed a full 1920×1080 screenshot per step in the trace, so a single rollout's sample exceeds 25 MiB and gets dropped on
--push— leaving those runs with no hosted rollout link. Observed while validating thecua_world_v1/osworld_v2_v1research environments:sample 0 is too large to upload (53564890 > 26214400 bytes)— cua_world, kimi-k3 (~51 MiB @ 15 turns)sample 0 is too large to upload (78391638 > 26214400 bytes)— osworld_v2, qwen (~75 MiB @ 50 steps)At ~1.5–3.5 MiB per screenshot, one rollout can run well past 25 MiB, and longer/higher-res computer-use rollouts go higher still (a 50-turn cua rollout would be ~170 MiB). Raises
max_payload_bytes25 → 100 MiB in both the sync and asyncpush_samples(the same two spots #637 touched) for real headroom without compressing traces.🤖 Generated with Claude Code