Licensing: record-and-track policy, and the per-dataset inventory for migration

[mmcky]

Inventory of datasets migrating into this repo whose redistribution status needs a decision, and the record of the policy that governs them. Opened 2026-08-06 so the migration can proceed without stalling per-dataset on licence questions, while nothing goes unrecorded.

This is the per-dataset working inventory for this repo; QuantEcon/workspace-lectures#20 remains the org-level thread where an item's fate (confirm / replace / remove) is decided.

The policy

Licensing does not gate migration. These datasets are already served publicly from the lecture repos — in several cases for years. Moving the same bytes to a canonical host with better provenance, attribution and an explicit licence field is a strict improvement on the status quo, and holding the migration for licence review would trade a real, present reproducibility problem for a hypothetical one.

So the rule is record and track, not block:

1. Migrate what we already serve. The manifest records the licence as found, including redistribution: restricted and name: null where that is the honest answer.
2. Inherit the existing status. A file the lectures have served for years arrives with that fact recorded, not with a fresh legal review as a precondition.
3. Track anything needing further thought here, with alternatives, and resolve it before data.quantecon.org is promoted as a public open-data endpoint — that promotion, not the migration, is the real gate.

This generalises the exception AGENTS.md already carries for countries.csv into the default for inherited data. It does not extend to new datasets: a dataset arriving here for the first time, with no prior life in a lecture repo, should still have its licence established before it lands — as the P5 additions all did.

Already recorded — the ten migrated datasets

Eight are cleanly permitted and need nothing further. Two are listed below as open items.

Dataset	Licence as recorded	Status
employ.csv	Eurostat reuse (Commission Decision 2011/833/EU)	permitted
epl_match_goals.csv	Public domain	permitted
japan_deaths_by_age.csv	CC BY 3.0 IGO	permitted
japan_earthquakes.csv	US Government work — public domain	permitted
japan_population_by_age.csv	Japan Statistics Bureau terms of use	permitted
lingcod_msy_recovery.csv	CC BY 4.0 (established against the Zenodo DOI record)	permitted
realwage.csv	CC BY 4.0	permitted
us_adult_heights.csv	US Government work — public domain	permitted

Open items

1. countries.csv — restricted, migrated, in use

WorldData.info, "all rights reserved", no redistribution grant stated. Already redistribution: restricted in its manifest and already tracked at QuantEcon/workspace-lectures#20. Consumed by pandas_panel in both lecture-python-programming and lecture-python.myst, where it supplies a country-to-continent lookup — 2 of its 17 columns.

Alternatives: an openly-licensed country-to-continent mapping would replace the only part the lectures use, at the cost of re-validating figures. This is the strongest candidate for replacement rather than permission-seeking, because the substitute is trivial and abundant.

2. ames_house_prices.csv — no formal licence, reasoned as permitted

Recorded name: null, redistribution: permitted, verified: 2026-08-03, with the reasoning in the manifest: the source article states no formal licence but presents the data as a teaching set published for reuse, distributed from the journal's own server. Long-established redistribution in practice.

No action expected — listed so that "the licence field is empty" is not repeatedly re-discovered as a gap. It is a documented decision, not an omission. (The 2026-08-04 migration plan lists this as an open manifest gap; that entry is stale.)

To assess as they migrate

None of these blocks its wave. Each gets redistribution recorded at migration time; the ones that come back anything other than a clear yes get a row in the Open items table above.

Dataset(s)	Source	Expectation
forbes-global2000.csv, forbes-billionaires.csv	Forbes, scraped via the Forbes API (webscrape_forbes.ipynb)	most likely to be restricted — commercial publisher, scraped. Treat as the countries.csv case if so
cities_us.csv, cities_brazil.csv	worldpopulationreview.com	unestablished; same publisher class as WorldData.info, so expect to need a decision
SCF_plus_mini.csv, SCF_plus_mini_no_weights.csv, SCF_plus.dta	Kuhn–Schularick–Steins replication data	published replication data — normally permitted, verify against the deposit record rather than a homepage
maketable1/2/4.dta	AJR 2001 replication	published replication data
fp.dta	Treisman 2016	published replication data
dataBHS.mat	BHS MATLAB replication bundle	published replication data
bbh_macro_quarterly.csv, bbh_michigan_monthly.csv	Zenodo replication package	already known CC BY 4.0
acs_data_summary.csv	American Community Survey	US Government work — expect public domain
mpd2020.xlsx	Maddison Project 2020	expect CC BY 4.0 — confirm against the release page
life-expectancy-vs-gdp-per-capita.csv	Our World in Data	expect CC BY
test_pwt.csv	Penn World Table 7.0 extract	expect permitted
NEWQDATA.csv	Cogley–Sargent 2005	published dataset
fred_data.csv	FRED (GS1/GS5/GS10/DFII5/DFII10/USREC)	check the underlying series — AGENTS.md warns that FRED re-serves third-party series that may not be redistributable. These six look like Treasury/NBER series, which are fine, but confirm
hansen_singleton_1982/1983_data.csv, hansen_jagannathan_1991_data.json	constructed from documented sources	inherits its sources' terms
longprices.xls, chapter_3.xlsx, assignat.xlsx, dette.xlsx, fig_3.xlsx, caron.npy, nom_balances.npy	author-assembled — QuantEcon's own transcriptions from cited publications	record as such; the transcription is QuantEcon's own work

The real gate

Resolve every Open item before data.quantecon.org is announced as a public open-data host. That is a decision point in PLAN.md Phase 4 and #15, not a per-migration checkpoint.

Related: QuantEcon/workspace-lectures#20 (org-level licensing thread), #15 (URL cutover — the promotion gate), QuantEcon/meta#338 (pilot), QuantEcon/QuantEcon.manual#108 (convention). Part of #8.

[mmcky]

The six high_dim_data files are the next wave to arrive (P3 in migration.yml), so I went through their upstream provenance. The licence expectations in the assessment table all hold. Two things do not, and both need deciding before the fold PR is written rather than at migration time.

1. SCF+ has no existing status to inherit

Policy rule 2 says a file arrives with its recorded status, not a fresh review. For SCF+ there is nothing recorded to carry over. SCF_plus/README.md (4,974 B) is a variable dictionary — a header, four lines on units and imputation implicates, then a 60-row variable/label table. It contains zero URLs, zero dates, no licence, no DOI, and the word "source" does not appear anywhere in it. The provenance table lives only in cross_section/README.md (2,173 B), which covers the four cross-section files and says nothing about SCF+.

So the SCF row needs a positive decision, and there are two honest ones. Either establish it against the Kuhn–Schularick–Steins (JPE 2020) deposit record — which is what the table already asks for, and which needs someone to match the deposit to these bytes — or, if that match cannot be made, land retrieved: null, license.name: null and integrity.upstream.status: unverifiable rather than inventing a value. Specifically: do not use the high_dim_data commit date as retrieved. That dates our copy, not the retrieval.

Two related mechanics. The minis are derived, not retrieved: SCF_plus/generating_mini.md reads SCF_plus.dta, keeps 7 of its columns and renames them; both to_csv calls are commented out upstream, so the committed builder writes nothing while the fetch and transform are complete. And SCF_plus.dta appears in the table above but not in P3's dataset list — no lecture reads the .dta directly, only generating_mini.md does. Whichever way that goes, the deposit-record question still has to be answered, because both minis inherit their record from it.

2. webscrape_forbes.ipynb is a separate row from the Forbes data

The table treats the two CSVs and their builder as one item. They are two different questions, and they need not resolve the same way. Caching data we already serve is the policy's case. Republishing the scraper from a canonical, publicly-served dataset repo is not.

The global2000 cell requests https://www.forbes.com/forbesapi/org/global2000/2020/position/true.json?limit=2000 — an undocumented internal endpoint, still live — with a spoofed Chrome 87 user-agent, a matching referer, and hardcoded GDPR consent cookies (notice_behavior: "expressed,eu", notice_gdpr_prefs). The billionaires cell hits a second undocumented endpoint, http://www.forbes.com/ajax/list/data. That is a terms-of-service question about the method, independent of whatever redistribution the CSVs get. The data can land restricted-and-tracked while the notebook either does not come across at all, or comes across with the consent-cookie and user-agent spoof stripped and a note that it no longer runs as-is.

The six, with evidence

Dataset	Source as recorded upstream	Expectation
forbes-global2000.csv, forbes-billionaires.csv	forbes.com/lists/global2000/, forbes.com/billionaires/ — recorded in cross_section/README.md, no grant of any kind	restricted, as the table says. Treat as the countries.csv case
cities_us.csv, cities_brazil.csv	worldpopulationreview.com/us-cities, /countries/cities/brazil	restricted expected. Like countries.csv, the substitute is abundant — both are plain population-by-city tables
SCF_plus_mini.csv, SCF_plus_mini_no_weights.csv	nothing recorded; derived from SCF_plus.dta	needs the deposit record or an explicit unverifiable record, per above

Two files the inventory does not cover

Both are still to migrate and appear in neither the Open items nor the assessment table. usa-gini-nwealth-tincome-lincome.csv is a second-order derivative — _static/lecture_specific/inequality/data.ipynb:100 writes it from SCF_plus_mini — so it inherits whatever record SCF+ gets and cannot be assessed ahead of it. graph.txt is QuantEcon's own generated output (short_path.md:272 writes it with %%file, 100 nodes), so it belongs in the author-assembled row with longprices.xls and chapter_3.xlsx.

None of this blocks the wave, consistent with the policy. The SCF+ record is the one item that has to be chosen rather than copied, and it has to be chosen while the manifest is being written.