evalkit — evaluate language models the right way

Most LLM eval reports are statistically broken: a bare accuracy number with no confidence interval, an LLM judge with uncontrolled position bias, "model A beats model B" with no significance test, and no check for whether the test set leaked into training. evalkit is a drop-in skill that gives any coding agent (Claude Code, Cursor, Codex, …) the tools to evaluate models correctly — and the statistics core is pure-numpy, installs instantly, and is validated against ground truth.

You:   Model A scores 82% and model B scores 79% on my benchmark. A is better, right?

Agent: [runs `evalkit compare ...`]
       Not necessarily. The gap is +3.0% with a 95% CI of [-1.5%, +7.5%] — it
       includes zero — and McNemar's test gives p = 0.286. On these 200 items
       the difference is within noise. You'd need ~5x the data to call it.

Why this exists

Ask a coding agent to "evaluate this model" and it will happily report accuracy: 0.82 and call it a day — no interval, no significance test, no bias check. That is how wrong conclusions get shipped. evalkit packages the parts of evaluation methodology people skip, behind one CLI and a SKILL.md that forces the agent to report uncertainty and test significance before claiming a winner.

See it in 10 seconds (no model, no API, no downloads)

./run.sh setup && ./run.sh demo

Four self-contained acts, each validating a method and showing the failure it prevents:

Act	Shows	Result
1. Naive vs. correct	"A beats B" is often noise	gap CI [-1.5%, +7.5%], McNemar p=0.286 → not significant
2. Calibration	the bootstrap CI is trustworthy	nominal 90% → empirical 91%, 95% → 94% coverage
3. pass@k	the unbiased estimator	pass@1=0.20, pass@5=0.78, pass@10=1.0 for (n=10, c=2)
4. Judge bias	position bias is caught	a first-position-biased judge → position_bias_rate=1.00, 0 trustworthy decisions

What it does

Command	Technique	Question it answers
eval	log-likelihood MC + bootstrap CI	How accurate is this model — with what uncertainty?
compare	McNemar + paired bootstrap diff CI	Is A significantly better than B?
passk	unbiased pass@k (Chen et al. 2021)	What's the real pass@k for my code samples?
judge	swap-order LLM-as-judge	Which output wins, after removing position/verbosity bias?
contamination	n-gram overlap (n=13, GPT-3 style)	Did my test set leak into the reference corpus?
demo	all of the above	Why does any of this matter?

Add --json to any command for machine-readable output (what the agent parses).

Validated against ground truth

evalkit's correctness contract — ./run.sh test checks it every run (pure numpy, ~3s):

Method	Validated by
pass_at_k	matches the analytic 1 − C(n−c,k)/C(n,k) exactly
bootstrap_ci	calibrated: 90% CI achieves ~90% empirical coverage in simulation
mcnemar_exact	matches known exact-binomial values (e.g. b=10,c=0 → p=0.00195)
wilson_ci	stays informative at 100% accuracy where the bootstrap collapses
position-bias detection	recovers an injected biased judge (rate → 1.00)
contamination	flags verbatim overlap, ignores unrelated text

$ ./run.sh test
........                                                  8 passed in 3.05s

Evaluate a real model

./run.sh setup-hf        # adds torch + transformers

# multiple-choice accuracy with a confidence interval
evalkit eval examples/capitals.jsonl --model distilgpt2
#   distilgpt2 on 12 items
#   accuracy = 0.8333 [0.5833, 1.0000] (95% CI)

# is gpt2 significantly better than distilgpt2 here?
evalkit compare examples/capitals.jsonl distilgpt2 gpt2

Tasks are JSONL: {"question": "...", "choices": [...], "answer": <index>}. Multiple choice is scored by log-likelihood (the deterministic, generation-free method used by lm-eval-harness), so results reproduce exactly on CPU.

For judging and generation, point --base-url at any OpenAI-compatible endpoint (OpenAI, vLLM, Ollama, LM Studio) and set EVALKIT_API_KEY.

Use as a library

from evalkit import bootstrap_ci, pass_at_k, mcnemar_exact, paired_diff_ci

bootstrap_ci([1,0,1,1,0,1,1,1,0,1])        # 0.70 [0.40, 0.90] (95% CI)
pass_at_k(n=10, c=2, k=5)                   # 0.7778
mcnemar_exact(b=14, c=8)                    # 0.286  (A vs B not significant)

Honest limitations

• CIs quantify sampling noise, not validity. A tight CI on a contaminated or biased benchmark is precisely wrong — run contamination and report it.
• Log-likelihood MC scoring favors base models and is sensitive to choice phrasing/length; use --normalize (acc_norm) when options differ in length.
• LLM judges have biases beyond position/verbosity (self-preference, style). evalkit controls the two biggest and measures rather than hides the rest.
• McNemar needs paired data (same items, both models); for unpaired comparisons use the bootstrap difference CI.

Citation

@software{bano2026evalkit,
  author = {Bano, Azra},
  title  = {evalkit: rigorous LLM evaluation as an agent skill},
  year   = {2026},
  url    = {https://github.com/azrabano23/evalkit}
}

Companion to interp — understand a model's internals, then measure its behavior rigorously.

License

MIT — see LICENSE.