Skip to content

Impute capital gains amounts from the published joint distribution - #560

Open
MaxGhenis wants to merge 7 commits into
uk-cgt-source-stagefrom
uk-cgt-imputation
Open

Impute capital gains amounts from the published joint distribution#560
MaxGhenis wants to merge 7 commits into
uk-cgt-source-stagefrom
uk-cgt-imputation

Conversation

@MaxGhenis

Copy link
Copy Markdown
Contributor

Part two of #552, stacked on #558 (the source surface). Review #558 first; this PR's own diff is the last commit.

What the stage does

Redraws existing gainers' capital_gains from HMRC table 3 — the published joint distribution of size of gain by taxable income — so the population can hold the gains above £2m that carry most of the tax. The measured gap this closes: the two top bands hold 54.6% of UK CGT and effectively none of the modelled tax, because the incumbent percentile source stops near £1m.

The stage owns amounts only. Incidence comes from the candidate, household weights pass through untouched (no mass change to record), and whether the published band facts also become calibration targets remains the fence adjudication recorded on the issue — nothing here assumes an answer.

Design

  • Conditioning. HMRC's rows condition on taxable income after reliefs and the Personal Allowance, which the raw-input population does not carry. The proxy sums the persisted components of the model's total_income concept (ITA 2007 s.23) and subtracts a tapered allowance, with the allowance amounts read from the policyengine-uk parameter tree at a stated instant — the uk extra's deferred-import pattern, per its own comment that the base package never imports policyengine-uk at import time. state_pension_reported stands in for social_security_income; reliefs are not deducted. Both divergences are documented where they can move a person one band.
  • Allocation. Rank-preserving within income band: the largest existing gainers absorb the published taxpayer mass, highest gain band first, scaled proportionally where the population holds less gainer mass than HMRC's taxpayers. No person splits across bands, so band mass is matched to one household weight's granularity.
  • Amounts. Within-band distributions matched to each cell's published mean — table 3 publishes a count and an amount per cell, so the mean is data, not assumption. Bounded bands use a truncated exponential (the one-parameter family whose mean can sit anywhere in the band, degenerating to uniform at the midpoint); the open £5m+ band uses a Pareto whose shape the published mean pins. A cell whose count is suppressed falls back to its band-total mean — dividing a published amount by an assumed count can put the implied "mean" outside the band, which is exactly the trap the band totals added in Read HMRC's joint distribution of capital gains by taxable income #558 exist to avoid.
  • Sub-AEA gainers. Table 3 covers only individuals with a CGT liability, so gainers beyond the published taxpayer mass keep their existing amounts capped at the annual exempt amount — not invented into the liability distribution, not deleted. Stated as scope in the module docstring.
  • Determinism. Draws are seeded on the build period; two runs are bit-identical, two periods differ.
  • Reporting. summarize_uk_cgt_imputation compares achieved band totals with the published surface. It is reporting, not a gate: where the population holds less gainer mass than HMRC's taxpayers the achieved totals sit below published by construction, and holding levels to the surface is the calibration question.

What it deliberately does not do

No family_coverage entry, no cgt_source_stages.json, no band-level UK_CGT_TARGET_SPECS. Promoting the stage to required_at_build and declaring targets follow the fence adjudication on #552, on the individuals-only basis (the two existing facts include trusts).

Tests

17 new tests: proxy arithmetic including the taper, mean-matching for both draw families (with the removable singularity at the uniform midpoint covered), determinism, rank preservation, the sub-AEA remainder, suppressed-cell fallback, artifact verification before read, and the parameter-tree read (runs when the uk extra is installed; importorskip otherwise — it is absent from the secrets-free PR gate by design). Full populace-build suite green locally with the extra installed.

🤖 Generated with Claude Code

The stage redraws existing gainers' amounts from HMRC table 3 so the
population can hold the gains above £2m that carry most of the tax. It owns
amounts only: incidence comes from the candidate, household weights pass
through untouched, and whether the published band facts also become
calibration targets stays a separate adjudication.

Taxable income is an arithmetic proxy — the persisted components of the
model's total_income concept less a tapered Personal Allowance, with the
allowance amounts read from the policyengine-uk parameter tree at a stated
instant through the uk extra's deferred import, following the US runtime's
pattern. Within each income band the allocation is rank-preserving: the top
of the existing ranking absorbs the published taxpayer mass, highest gain
band first, scaled to the available mass where the population holds less
than HMRC's taxpayers. Amounts then come from within-band distributions
matched to each cell's published mean — truncated exponential on bounded
bands, Pareto on the open top band — with a suppressed cell falling back to
its band-total mean. Gainers beyond the taxpayer mass keep their existing
amounts capped at the annual exempt amount rather than being invented into
the liability distribution or deleted.

Draws are seeded on the build period, so a build is reproducible and two
periods draw differently. A summary helper reports achieved band totals
against the published surface; it is reporting rather than a gate, since
holding levels to the surface is the calibration question.

Part two of #552, on top of the source surface in #558.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator

Heads-up on a planned collision: #612 (carrying UK build datasets as populace Frames) is in motion — the foundations PR #617 is up, and the carrier-swap PR will stack on #610 and re-type UKNationalStage.transform to Frame -> Frame. This PR's cgt_imputation.py transform is typed on UKNationalDataset (and constructs it directly, plus the _dataset() test helper), so whichever lands second pays a small mechanical rebase.

Both orders are pre-agreed fine: if this merges first, the swap absorbs the conversion as one more stage (same pattern as the two production stages); if the swap merges first, I'm happy to push the Frame conversion onto this branch myself so it doesn't sit in your queue. No hold needed either way.

MaxGhenis and others added 2 commits August 7, 2026 12:04
The carrier swap (#618, increment 1 of #612) retyped UKNationalStage to
Frame -> Frame and retired UKNationalDataset, landing after this stage was
written — the collision flagged on the PR, paid here as the mechanical
adaptation: read through frame.table, resolve the build period from frame
metadata, and rebuild through uk_national_frame with the kind and mass log
carried unchanged, the migrated stages' person-only-replacement idiom.

The adaptation exposed a coverage gap: no test ran the stage factory's own
transform, so a retired type surviving in its signature was invisible under
postponed annotation evaluation. A new test runs the stage end-to-end on the
pinned artifact where it is present locally.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@MaxGhenis

MaxGhenis commented Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

The swap landed first, so this branch paid the rebase as predicted: 1470a33 merges main, and the adaptation commit carries the stage on the Frame carrier — reads through frame.table, build period from frame metadata, rebuild via uk_national_frame with kind and mass log unchanged (the person-only-replacement idiom from the migrated stages). The adaptation exposed that no test ran the stage factory's own transform (postponed annotations hid a retired type in its signature), so there's now an end-to-end test on the pinned artifact. All 46 CGT tests green; full suite running.

MaxGhenis and others added 2 commits August 7, 2026 14:13
Adversarial review against the published table found the synthetic test
surface masking three defects (microcosm#560 review):

Published values round to the nearest thousand people and million pounds,
and four cells of the 2023-24 table imply a mean outside their own band —
£463m over a rounded 1,000 people reads as £463,000 for the £500k-£1m band.
The solvers refused those means, so the stage crashed on any population
reaching four of the six income bands, while every synthetic test mean sat
comfortably in band and the one real-artifact test used the single income
band with no infeasible cell. Implied means are now repaired to just inside
the violated boundary, keeping the signal that the cell's mass sits near
that edge.

Suppressed-count cells were allocated a fixed 500 people each, overstating
tail taxpayer mass against the published totals. They now split the
residual between the published income-column total — newly captured from
the All row — and the column's unsuppressed cells, so allocation reconciles
to published numbers with no invented count.

The bottom band spans the annual exempt amount, so its draws could land at
or below the AEA: people allocated as taxpayers with a liability then read
as non-liable downstream. Its support is now floored at the AEA where that
fits inside the band.

The published 2023-24 surface is embedded in the tests as literals — public
statistics — so CI exercises the real rounding pathology without the
artifact, and the pinned-artifact end-to-end test now spans all six income
bands.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

An adversarial review pass (cross-model, pre-merge) blocked this with four criticals, all confirmed against the published table. Fixed on the branch; the review trail:

C1 — rounded means outside their band crash the stage. Confirmed: four cells of the 2023-24 table imply a mean below their band floor through rounding — £463m over a rounded 1,000 people reads as £463,000 for the £500k–£1m band (also (£2m–£5m, £50k), (£1m–£2m, £100k), (£250k–£500k, £150k)). The solvers refused those means, so the stage crashed for any population reaching four of the six income bands — while every synthetic test mean sat feasibly in band, and the one real-artifact test used the single income band with no infeasible cell. Implied means are now repaired to just inside the violated boundary (2% of band width), keeping the signal that the cell's mass sits near that edge.

C2 — the 500-person suppression convention overstates mass. Replaced with reconciliation: suppressed cells split the residual between the published income-column total (newly captured from the All row, on #558) and the column's unsuppressed cells. No invented count survives; allocation reconciles to published totals by construction.

C3 — bottom-band draws at or below the AEA. The £0–£10k band straddles the £6,000 AEA, so draws could land people allocated as liable taxpayers below the liability threshold. Its support is now floored at the AEA where that fits inside the band.

C4 — the masking. The real 2023-24 surface is now embedded in the tests as literals (public published statistics), so CI exercises the exact rounding pathology with no artifact present — including a test pinning the repaired £500k-band mean — and the pinned-artifact end-to-end test spans all six income bands instead of the one clean one.

Cleared by the same review: deterministic RNG/ranking, and the Frame weight-kind/mass/linkage invariants.

MaxGhenis and others added 2 commits August 7, 2026 14:22
…percase

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants