UK scorecard: populace_uk_2023 vs enhanced_frs_2024_25 vs admin benchmarks (2026)

[vahid-ahmadi]

cc @juaristi22 — María, could you take a look at why we're off on some of these?

I ran a benchmark scorecard of populace_uk_2023 against enhanced_frs_2024_25 (both simulated for 2026 through policyengine.py, pe.uk.ensure_datasets → Simulation with the pinned UK model version) and compared to approximate admin benchmarks.

Scorecard (2026)

Metric	populace_uk_2023	enhanced_frs_2024_25	Benchmark (~2026)
Population	69.9m	70.0m	ONS ~68.3–69m
Households	29.5m	31.5m	ONS ~29m
State pension	£130.8bn	£127.5bn	OBR ~£140–146bn
Income tax	£419bn	£312bn	OBR ~£330bn
Income taxpayers	36.7m	42.7m	HMRC ~37–40m
Universal credit	£39.9bn / 3.3m families	£81.3bn / 6.35m	~£80–86bn / ~6.7m hh
Child benefit	£17.4bn	£17.7bn	~£13.5bn
Pension credit	£6.8bn	£7.5bn	~£6bn
Council tax	£60.1bn	£52.5bn	~£50bn
Poverty (BHC)	16.7%	13.5%	HBAI ~17%
Top 1% share (net hh income)	6.9%	6.9%	—
Gini (net hh income)	0.385	0.370	ONS ~0.34–0.36

Populace does well on demographics, household counts, taxpayer counts, pension credit, and poverty. The two large misses are:

1. Income tax +27% vs OBR (£419bn vs ~£330bn) — consistent with the +20.8% vs the SPI-anchored benchmark already diagnosed in UK_COVERAGE_PROGRESS.md as distributional.
2. Universal credit at less than half of admin (£39.9bn / 3.3m families vs ~£80–86bn / ~6.7m households) — consistent with Certified UK artifact carries less than half of admin UC caseload (3.26m vs ~6.7m; £36.9bn vs £75.8bn) #701 (UC not bound in the certified 148-target surface).

Smaller ones worth a look: child benefit ~+30% (though enhanced FRS shares this), council tax ~+20%, state pension ~£10–15bn low (also shared), and Gini a touch high.

Reproduction code

"""Benchmark scorecard for a UK dataset via policyengine.py (year 2026).

Usage: python scorecard.py <dataset-name-or-hf-uri>
e.g.   python scorecard.py populace_uk_2023
"""

import sys

import numpy as np
import policyengine as pe
from policyengine import Simulation

DATASET = sys.argv[1]

d = pe.uk.load_datasets(datasets=[DATASET], years=[2026], data_folder="./data")
dataset = list(d.values())[0]

PERSON_VARS = [
    "age",
    "state_pension",
    "income_tax",
    "national_insurance",
    "employment_income",
    "self_employment_income",
    "dividend_income",
    "pension_income",
    "capital_gains",
]
BENUNIT_VARS = ["universal_credit", "child_benefit", "pension_credit"]
HH_VARS = ["household_net_income", "income_tax", "council_tax", "in_poverty_bhc"]

sim = Simulation(
    dataset=dataset,
    tax_benefit_model_version=pe.uk.model,
    extra_variables={
        "person": PERSON_VARS,
        "benunit": BENUNIT_VARS,
        "household": HH_VARS,
    },
)
sim.ensure()
out = sim.output_dataset.data


def arr(ent, var):
    return np.asarray(getattr(out, ent)[var], float)


pw = arr("person", "person_weight")
bw = arr("benunit", "benunit_weight")
hw = arr("household", "household_weight")

print(f"SCORECARD {DATASET} year 2026")
age = arr("person", "age")
print(f"population_m {pw.sum()/1e6:.2f}")
print(f"children_u16_m {pw[age < 16].sum()/1e6:.2f}")
print(f"age_65plus_m {pw[age >= 65].sum()/1e6:.2f}")
print(f"households_m {hw.sum()/1e6:.2f}")

for v in PERSON_VARS[1:]:
    print(f"{v}_bn {(arr('person', v)*pw).sum()/1e9:.1f}")
it = arr("person", "income_tax")
print(f"taxpayers_m {pw[it > 0].sum()/1e6:.2f}")

for v in BENUNIT_VARS:
    x = arr("benunit", v)
    print(f"{v}_bn {(x*bw).sum()/1e9:.1f}")
    print(f"{v}_families_m {bw[x > 0].sum()/1e6:.2f}")

hni = arr("household", "household_net_income")
print(f"council_tax_bn {(arr('household','council_tax')*hw).sum()/1e9:.1f}")
pov = arr("household", "in_poverty_bhc")
print(f"poverty_bhc_pct {100*(pov*hw).sum()/hw.sum():.1f}")

order = np.argsort(hni)
h, w = hni[order], hw[order]
cw = np.cumsum(w) / w.sum()
inc = h * w
print(f"top1_share_pct {100*inc[cw >= 0.99].sum()/inc.sum():.1f}")
print(f"top10_share_pct {100*inc[cw >= 0.90].sum()/inc.sum():.1f}")
p = np.cumsum(w) / w.sum()
L = np.cumsum(np.clip(inc, 0, None)) / np.clip(inc, 0, None).sum()
print(f"gini {1 - 2*np.trapezoid(L, p):.3f}")

Notes: national_insurance here is the personal-side variable only; benchmarks are approximate (OBR EFO / HMRC / DWP / ONS, nearest year) and meant for orders of magnitude, not precise scoring. All figures are weighted aggregates only — no microdata is included.

[juaristi22]

Thanks @vahid-ahmadi — dug into this. TL;DR: the pattern is exactly what the June release's calibration surface predicts. The certified artifact was calibrated to 149 targets: 142 ONS region×age population, 6 SLC student-support, and 1 OBR private-school-students (chronicle#132); the calibration code never reached a pushed branch and the national build is gate-only by design (#623). Zero fiscal targets bound — so demographics, household counts, taxpayer counts, and poverty land well, and every fiscal aggregate free-floats. The eFRS column is the mirror image: it binds OBR receipts + HMRC income bands, so it nails income tax while overshooting households (31.5m vs ONS ~29m) and distorting poverty low (13.5% vs HBAI ~17%). Each dataset is accurate exactly where its surface binds.

Row by row

Metric	Diagnosis	Fix path
Income tax +27%	Distributional, pre-exists uprating: +20.8% at base year (£334.6bn vs £277bn SPI-anchored, UK_COVERAGE_PROGRESS.md restoration diagnosis). Taxpayer count is right (36.7m); tax per taxpayer £11.4k vs OBR-implied ~£8.8k — upper-middle weight mass, unbound	obr.income_tax declared in the 186-target contract. Direct HMRC banded route is adjudicated-fenced (all 208 facts banded by SPI Total Income, which FRS can't materialize — hmrc_income_replay_report.json, calibration_performed: false)
UC −52%	#701. Purely caseload: per-unit entitlement matches admin exactly (£36.9bn/3.26m = £11.32k vs £75.8bn/6.7m = £11.31k). Two stacked causes: pre-uk-data#452 take-up vintage in the inputs (eFRS at that vintage: 4.19m), then demographic-only calibration eroding 4.19m → 3.26m	The full stack below — this slice lands first
Child benefit +30%	Shared with eFRS → not a populace-calibration defect. would_claim_child_benefit is a dataset take-up draw defaulting to True; both datasets simulate ~full entitlement while the ~£13.5bn benchmark is net of HICBC opt-outs (~700k families) + non-take-up	Take-up draw audit (model/data-shared); partly benchmark comparability — opt-out clawback returns as income tax
Council tax +20%	Same input columns as eFRS (extracted from the same enhanced-FRS pipeline), so weights-driven — consistent with the same upper-middle upweighting as income tax	18 VOA/scotgov band-stock targets + obr.council_tax declared
State pension low	Shared with eFRS → input-side: FRS under-reporting + survey universe excludes communal establishments	obr.state_pension declared
Gini 0.385	Partly basis: unequivalised household net income with negative-clipping isn't comparable to ONS ~0.34–0.36 (equivalised disposable). The populace-vs-eFRS delta (+0.015, same basis) is real — same signature as income tax	Follows the income-side binds
Population 69.9m	Likely a benchmark-vintage issue, not a data defect: the artifact binds ONS SNPP 2022-based projections, which put mid-2026 near 70m; ~68.3–69m looks like an earlier vintage	Re-base the benchmark row on SNPP 2022

How the in-flight stack closes this

The architecture the June release lacked — facts → declaration → activation → fail-closed solve — is landing as one chain, UC first:

1. chronicle#184 (in review now): facts-only DWP package — 11 monthly UC benefit-unit counts derived count÷share from deductions Table 1 (DWP "household" = UC assessment unit = benunit, uk-data#457). OBR UC expenditure facts already exist in chronicle. After merge: released consumer bundle + re-staged feed (currently zero uc_benefit_units facts).
2. Refresh the UK Universal Credit target contract #727: the value-free dwp.uc.households declaration + contract re-pin to uk-data 1.56.16. The contract (uk_national_targets.json, 186 declarations) already covers every missed family above — declarations resolve from chronicle facts only; the uk-data pin is parity accounting, not a value source.
3. Add the UK national calibration step over ledger-backed target references #729 (stacked on Refresh the UK Universal Credit target contract #727): the Calibrate the UK national build from Ledger-backed targets #623 calibration seam — activates UC caseload + OBR in-cap expenditure at 2025, fail-closed (an activated reference that can't compile aborts the build; the uk-data#456 silent-drop class is impossible by construction), with per-target rel-error/ESS diagnostics and a release-blocking gate. One adjudication we're adopting before the first real run: the monthly facts need a declared calendar-year-average operation in the RFC: the spec engine — schema for full spec-drivenness (review before build) #698 spec-engine vocabulary (alongside latest-plateau selection for 2026) — per Max's seam analysis on #701 — so the reference binds ~6.7m (2025 average) rather than silently resolving latest-month (Dec ≈ 7.17m, ~7% off the expenditure target's average-caseload concept; the contract's allowed_value_operations is ["identity"] today).
4. Consume Ledger UK facts: uk_target_references, mapping module, and a compile-parity gate #622 completion (in flight): full 186-declaration materialization + parity gate — the breadth wave that activates the income tax / council tax / state pension families after the UC slice proves the seam.
5. Retarget the UK build to FRS 2024-25: re-pin raw vintages, regenerate parity instruments, re-measure gate baselines #723: retarget to FRS 2024-25 (calibration year 2025) once E5 (Port the WAS wealth imputation and regional property uprating as declarative source stages #714) / E6 (Port the LCFS consumption, ETB VAT, and ETB services/NHS layers as declarative source stages #730) land — replaces the certified-2023-spine bridge with the right vintage, and the E-lane take-up rebuild raises the raw UC recipient count so calibration isn't doing all the lifting.

Then a gated build + certification, at which point this scorecard is a ready-made acceptance test — worth re-running the script as-is against the first calibrated candidate. The bar on UC is beatable: eFRS at 1.56.14 sits at 6.40m/£77.6bn vs admin ~6.7m/£75.8bn, and populace's per-unit entitlement already matches admin while keeping the demographic/household/poverty fidelity eFRS gives up.