Add live UK population microsimulation evals

[anth-volk]

Fixes #230

Summary

• add support for repeated tool-loop trials with pass-rate thresholds
• ground tool-loop final-answer numbers in executed tool outputs
• add 20 manual live/data-backed UK population microsimulation eval cases
• document a population-specific manual eval target

Tests

• PYTHONPATH=backend python -m pytest backend/tests/test_evaluation.py -q
• PYTHONPATH=backend python -m eval.run --mode offline --no-report

Notes

• make eval-ai-offline is blocked in this environment before the runner executes because the generated policyengine_uk tool-contract file is stale and asks to run make sync-policyengine-uk-evals.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
policyengine-uk-chat	Ready	Preview	Aug 6, 2026 4:45pm

[github-actions]

Beta preview has been cleaned up because this PR was closed.

[codecov]

Codecov Report

❌ Patch coverage is 75.92695% with 435 lines in your changes missing coverage. Please review.
✅ Project coverage is 62.54%. Comparing base (7519603) to head (7169d06).

Files with missing lines	Patch %	Lines
backend/chat/orchestrator.py	64.96%	68 Missing and 21 partials ⚠️
backend/gateway/assessment.py	63.92%	53 Missing and 26 partials ⚠️
backend/gateway/proposals.py	68.11%	46 Missing and 20 partials ⚠️
backend/eval/deployed_runner.py	68.62%	22 Missing and 10 partials ⚠️
backend/gateway/clarifications.py	66.66%	21 Missing and 9 partials ⚠️
backend/eval/run_deployed.py	0.00%	29 Missing ⚠️
backend/eval/collect_modal.py	0.00%	23 Missing ⚠️
backend/gateway/runtime.py	85.29%	5 Missing and 10 partials ⚠️
backend/eval/modal_batch.py	72.00%	7 Missing and 7 partials ⚠️
backend/chat/public_service.py	85.29%	5 Missing and 5 partials ⚠️
... and 13 more

❌ Your patch check has failed because the patch coverage (75.92%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #231      +/-   ##
==========================================
+ Coverage   57.66%   62.54%   +4.88%     
==========================================
  Files          82       99      +17     
  Lines        5088     6574    +1486     
  Branches     1045     1314     +269     
==========================================
+ Hits         2934     4112    +1178     
- Misses       1978     2188     +210     
- Partials      176      274      +98     

Flag	Coverage Δ
backend	81.54% <75.92%> (-1.01%)	⬇️
frontend	8.95% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
backend/api/main.py	72.34% <100.00%> (+1.22%)	⬆️
backend/chat/events.py	100.00% <100.00%> (ø)
backend/chat/system_blocks.py	76.66% <ø> (ø)
backend/eval/schemas.py	100.00% <100.00%> (ø)
backend/gateway/catalogue.py	96.07% <ø> (ø)
backend/prompts/__init__.py	100.00% <ø> (ø)
backend/prompts/gateway.py	100.00% <ø> (ø)
backend/tools/context.py	86.48% <100.00%> (+0.77%)	⬆️
backend/engine/reforms.py	75.30% <96.00%> (+7.12%)	⬆️
backend/gateway/execution.py	98.11% <98.11%> (ø)
... and 21 more

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.