How to choose --timeout-multiplier for diffenrent models on the official leaderboard?

[Maokui-He]

Hi, I noticed that some benchmark runs use --timeout-multiplier. Is there a recommended --timeout-multiplier setting per model, or a guideline for choosing the right value? For example, for slower/larger models, what multiplier do you suggest? Does the official leaderboard use a fixed multiplier across all models, or does it vary?

[ScuttleBot]

👋 Hi @Maokui-He! I'm @olearycrew's OpenClaw bot, helping triage this repo.

Good question — this comes up often!

Official Leaderboard: Fixed Multiplier

The official leaderboard uses --timeout-multiplier 1 (the default) for all models. No per-model adjustment. The reasoning: timeout behavior is part of what the benchmark measures. A model that needs 3x the time to complete a task has a real-world cost in user experience.

Guidelines for Community Runs

If you're testing locally or with slower infrastructure:

Scenario	Suggested Multiplier
Hosted API (OpenRouter, Anthropic, OpenAI)	1 (default)
Large local model (70B+, single GPU)	2-3
Quantized model on consumer hardware	3-4
Very slow inference (e.g., CPU-only)	4-5

How It Works

Each task has a timeout_seconds value in its frontmatter (typically 120-300s). The multiplier scales this linearly:

actual_timeout = task.timeout_seconds × timeout_multiplier

So --timeout-multiplier 2 on a 180s task gives 360s.

Important Note

If you use a multiplier > 1, your submission will still be accepted on the leaderboard, but the actual time taken is recorded. The score reflects task success, not speed — but official comparisons assume the default timeout window.

For the most comparable results to the leaderboard, use --timeout-multiplier 1. Only increase it if you're seeing timeouts that you believe are due to infrastructure (not model capability).

Feel free to ignore if this isn't helpful!