Proposal: a single canonical QuantEcon datasets repository, with a documented convention

Floating this for discussion. If there's rough agreement it should graduate to a QEP

Summary

Lecture data is currently hosted and referenced in at least four inconsistent ways across our lectures, with no guidance in the manual.

This causes frictions — see below.

I propose we adopt one canonical, general-purpose data repository, referenced by stable URLs, and document the convention in the manual.

Concretely: repurpose/rename QuantEcon/high_dim_data → QuantEcon/datasets rather than start a second repo.

Motivation

1. New-data files can't build on a branch (today's trigger).

While adding a population-by-age figure to prob_dist.md, the cell read the .xlsx via a https://github.com/QuantEcon/lecture-python-intro/raw/main/… URL — the pattern used by long_run_growth.md etc. Because the file was new and not yet on main, the local/PR build 404s and fails under CI's -nW. The only ways through are a two-step merge (land the data in one PR, then reference it in a second) or temporarily swapping the URL to a local path for the build. Both are fiddly and undocumented, and every contributor rediscovers this.

2. Four+ inconsistent data-hosting patterns, no rule. In lecture-python-intro alone:

raw/main URL from lectures/datasets/ — mpd2020.xlsx, longprices.xls, chapter_3.xlsx
raw/main URL from lectures/_static/lecture_specific/ — inequality Gini CSV, simple_linear_regression
relative local path, no URL — msy_fishery.md: pd.read_csv('datasets/lingcod_msy_recovery.csv')
external repo high_dim_data via two different hosts — plain raw.githubusercontent.com and media.githubusercontent.com (LFS)

There is no documented rule for which to use, so the choice is made ad hoc per lecture.

3. A lecture pinned to a data branch. mle.md reads from
…/high_dim_data/update_scf_noweights/SCF_plus/… — a feature branch, not main. If that branch is ever deleted or rebased, the lecture silently breaks. (Worth fixing regardless of this proposal.)

4. The data repo is mis-scoped. high_dim_data's README says "Data files for High Dimensional Economics." But its actual consumers are the intro lectures inequality, heavy_tails, mle (SCF wealth data, Forbes 2000/billionaires, US/Brazil city populations). It is already our de-facto general empirical-data repo — just misnamed, with a stale README.

5. Binary data bloats lecture repos. Committing .xlsx/.csv into the lecture repos (and into _static/) mixes binaries with narrative source. A dedicated data repo keeps lecture repos lean and text-focused.

6. No manual guidance. The style guide has zero coverage of data handling — the root cause of the inconsistency above.

7. Cross-series reuse. The same datasets (Maddison, SCF, …) are candidates for reuse across intro / programming / advanced / jax / julia. A shared data repo enables reuse instead of per-repo copies.

8. Colab / download self-containment. Relative paths break when a notebook is downloaded or opened in Colab; a stable external URL keeps notebooks self-contained. This is the legitimate reason to prefer URLs — and a dedicated data repo makes those URLs always resolvable regardless of the lecture branch, which is exactly what removes motivation #1.

Proposal

Adopt a single, dedicated, general-purpose data repository as the canonical home for lecture datasets, referenced by stable URLs, with the convention documented in the manual. Specifically, repurpose high_dim_data into QuantEcon/datasets, rather than create a second general repo — two general-purpose data repos would reintroduce the "which repo?" confusion.

Key design points:

LFS is per-path (.gitattributes): large binaries tracked via LFS, small teaching files stay plain git in the same repo.
Make it flat (just dump files) or organize by topic subfolders (demography/, inequality/, cross_section/, SCF_plus/, …).
One documented URL form and a decision on main-vs-tag pinning (see open questions).

Alternatives considered

Status quo (raw/main same-repo + two-step merges): ongoing friction, permanent per-lecture inconsistency.
Relative local paths everywhere (the msy_fishery pattern): simplest single-PR build, but breaks Colab/download self-containment and still bloats lecture repos.
Two repos split by "high-dim vs low-dim": the size boundary is fuzzy and technically unnecessary (LFS is per-path); the split is what causes confusion.
New datasets repo alongside high_dim_data: two parallel general repos → "which repo?" ambiguity. Since high_dim_data isn't really book-specific, repurposing it is cleaner.

Rollout

Confirm no external "High Dimensional Economics" project still depends on high_dim_data (nothing in the lecture series does — only intro's 3 files).
Rename QuantEcon/high_dim_data → QuantEcon/datasets. GitHub redirects preserve existing URLs, so this is non-breaking; the sweep below can follow at leisure.
Rewrite the README to state the real, general purpose.
Set .gitattributes for per-path LFS (large binaries LFS; small teaching files plain git).
Decide the layout (flat, or topic subfolders) and organize accordingly.
Standardize the URL form and pinning policy (open question below).
Sweep existing references to the new name (3 files in intro: inequality, heavy_tails, mle) and fix the mle.md branch-pin → main.
Document in the manual — new styleguide/datasets.md: the routing rule ("data owned by a specific book/project → that repo; otherwise → datasets"), LFS-per-path, the URL form, and the pinning policy.
Going forward: new lecture data → PR to datasets → reference by stable URL. A single lecture PR then builds green immediately.

Smaller issues to open regardless of the outcome

mle.md pins a data branch (update_scf_noweights) instead of main — latent breakage. (bug)
high_dim_data uses two URL hosts (raw vs media/LFS) for files in the same repo — standardize.

Open questions for the team

Repo name — datasets? data? Something else?
Layout — flat or by topic
Pinning — reference main (simple, mutable) or a tag/commit SHA (reproducible builds)?
URL form — settle plain raw.githubusercontent.com vs media.githubusercontent.com/LFS as a documented rule.

Recommendation on process

This touches every lecture series plus a data repo and establishes a working/tooling convention documented in the manual — per QEP-1 that is exactly what a QEP is for. Recommendation: use this issue to socialise and converge; once there's rough agreement, I'll graduate it to a QEP (type: standard) recording the datasets-repo convention, with styleguide/datasets.md as the implementation.

Context: this came out of a session adding a Japanese population age-distribution figure to prob_dist.md in lecture-python-intro.

@jstac

Thanks @jstac — I'm very much +1 on this, and the audit is spot on. The framing I'd add is that a dedicated data repo lets us cater to data needs directly, rather than piggy-backing on lecture infrastructure — data has its own lifecycle (provenance, refresh, licensing, size) and doesn't really belong in a narrative-source repo once it's used across multiple series.

Some history, one technical find that I think lets us sidestep your Q4 entirely, and thoughts on your open questions.

What exists today

I opened #129 for this ("DATA: Migrate high_dim_data to data") and we made a start: QuantEcon/data. The trigger was WASM/JupyterLite (#143) — pyodide can't fetch from most web APIs, so we needed snapshotted data at stable, CORS-friendly URLs. Worth adding to your motivation list, and it's why the repo has dynamic/ and scripts/ (periodic API snapshots) alongside static/.

A fair bit already landed in Feb 2025: intro's static data is already in QuantEcon/data today — mpd2020.xlsx, longprices.xls, chapter_3.xlsx, assignat.xlsx, dette.xlsx — via QuantEcon/data-lectures#5, plus a layout (QuantEcon/data-lectures#6) and a refresh script (QuantEcon/data-lectures#7). It then stalled on the last box of QuantEcon/data-lectures#4, "update all the paths in all the lectures", while we worked around the WASM issue. QuantEcon/data-lectures#1 (set up LFS) and QuantEcon/data-lectures#2 (fold high_dim_data in) are still open as well.

Q1 (name): `data-lectures`

I'd suggest we rename QuantEcon/data → data-lectures — data consumed by the lecture series.

It matches the taxonomy already documented in the manual's styleguide/repos.md — lecture-{name}, book-{name}, project-{name}, action-{name} are all {type}-{scope} — so your routing rule falls out of scope rather than size: consumed by lectures → data-lectures; owned by a specific book or package → that project's repo. It also leaves datasets free in case we ever want to host a curated QuantEcon dataset collection.

Q4 (URL form): a trap worth knowing about

The raw vs media split in high_dim_data isn't arbitrary — it's Git LFS. I tested all three hosts against the same LFS-tracked file and against a plain-git file:

URL form	LFS-tracked file	plain-git file
`raw.githubusercontent.com/...`	returns the LFS pointer text	works
`github.com/{org}/{repo}/raw/{ref}/{path}`	works	works
`media.githubusercontent.com/media/...`	works	404

This matters right now: QuantEcon/data has no LFS yet and holds mpd2020.xlsx (1.8MB) and dette.xlsx (632KB) as plain blobs. The moment we action QuantEcon/data-lectures#1, every raw.githubusercontent.com URL breaks silently by feeding pandas a pointer file — and that's the form intro uses today. (Also worth knowing: high_dim_data's .gitattributes is *.csv filter=lfs, i.e. every CSV regardless of size, so your per-path LFS point is a fix rather than just a setting.)

Which is why I'd propose `data.quantecon.org/lectures`

Rather than pick a GitHub host at all, let's serve the data ourselves via GitHub Pages on a custom domain — the same pattern as intro.quantecon.org. It gives the cleanest entry point for fetching data across every series, and sidesteps Q4:

The URL decouples from storage. Lectures reference https://data.quantecon.org/lectures/..., and LFS-vs-plain-git becomes internal hygiene that's invisible to every lecture.
CORS works — I verified Pages serves access-control-allow-origin: *, which is exactly what pyodide needs ([Plan/Idea] Supporting WASM based introductory series #143).
It fixes the LFS bandwidth problem, which is metered and billed — monitoring bandwidth usage high_dim_data#4 is literally "monitoring bandwidth usage". Readers hit Pages bandwidth rather than our LFS quota.
It's portable — a domain we own means we could move to R2/S3 later without touching a single lecture.

The /lectures prefix keeps the namespace open for /datasets or /books later without breaking any URL we ship now.

Two caveats for the QEP: the Pages deploy must checkout with lfs: true or we'd republish pointer files (our Pages builds are already build_type: workflow, so that's a one-line change); and Pages isn't a real CDN (~1GB site and 100GB/month soft limits), so worth monitoring.

data.quantecon.org already resolves — to an old NestJS box on AWS Sydney with no HTTPS listener — so I'll look into what's actually running there.

Q2 (layout): flat

I'd go flat, your first option. It's the one that really delivers your motivation #7: lecture-python and lecture-python-intro would both just fetch /lectures/mpd2020.xlsx, with no folder implying ownership. Topic folders would reintroduce a softer version of the consumer-keying problem the current data layout has.

The static/dynamic split then becomes metadata (provenance, refresh cadence) rather than directory structure — best surfaced in a generated index page at /lectures/, which doubles as the catalog. scripts/ stays in the repo, outside the published tree.

Q3 (pinning): `main`, with git history as the version record

Reference main. Git already gives us version control on the data, so we don't need a URL convention to reinvent it — a fix is just a commit, and if we ever need the old bytes they're a SHA away.

Two cases worth separating, though, because they look alike and want opposite handling. Corrections (bad parse, wrong units, corrupt rows) should be fixed in place: everyone should get the fix, and a new filename would leave the broken data live until we'd swept every reference. New vintages (Maddison 2020 → 2023, SCF 2019 → 2022) get a new filename, since that's genuinely different data rather than a new version of the same thing — the old vintage stays valid.

One consequence worth naming, since it couples back to Q4: serving from data.quantecon.org makes the published URL inherently "latest", so reproducing an old build becomes a git operation rather than a URL one. I think that's the right trade for continuously-rebuilt teaching material, but it's a deliberate one. The safety net is knowing what a correction touches — if the catalog records which lectures consume which dataset, a fix tells us exactly what to rebuild and review.

Process

Agree on the QEP, and the manual page is well overdue — it's the last unticked box on #129. One scope addition: styleguide/repos.md documents our repo-type taxonomy and has no data-repo type at all, so the QEP should add data-{scope} there, not just add styleguide/datasets.md.

Suggested consolidation: keep this issue as the design/QEP thread and let it absorb #129, with QuantEcon/data-lectures#1, QuantEcon/data-lectures#2 and QuantEcon/data-lectures#4 as a ready-made execution checklist.

+1 on both of your smaller issues regardless — the mle.md branch pin is live right now (update_scf_noweights is a real branch, and mle.md:93 still fetches from it, so it's latent rather than broken).

Happy to help pull the QEP together once we've converged.

prob_dist: motivating height example and Japan age distribution (#790)

Following up with data: I ran a full audit of dataset usage across all 7 Python-family lecture repos (each repo's main, 2026-07-15 — method: git grep for every reader call, data-file extension, GitHub host, and API client, cross-checked against git ls-tree inventories and the actual contents of QuantEcon/data and high_dim_data). Digest below; the two sub-issues (#337, #338) carry the execution side.

Headline numbers

Metric	Count
Distinct static data files referenced by lectures	34 (intro 17, programming 5, python.myst 9, advanced 6)
Data files committed across the repos	50, of which 20 are dead weight (orphaned, superseded, or shadowed by `%%file`)
GitHub raw-URL spellings in production	6 (not 4 — including `blob/…?raw=true` and two `refs/heads` variants)
Live references into the pre-MyST `lecture-python` repo (last push Jan 2022, unarchived)	8
Lectures fetching live API data	11 (wbgapi, FRED via three different access methods, yfinance)
Lectures referencing `QuantEcon/data`	0 — despite it already holding 10 intro files since Feb 2025

Only 4 of the 7 repos touch data at all: dp, jax, and continuous_time_mcs are data-free apart from an embedded %%file toy graph (which is itself duplicated in 3 repos).

Findings that sharpen the proposal

1. The mle.md branch pin is worse than latent. SCF_plus_mini_no_weights.csv does not exist on high_dim_data's main at all — the update_scf_noweights branch is the file's only home, so branch deletion is an instant 404. Tracked in #337.

2. Eight URLs depend on the dormant legacy repo, while both consuming repos already commit their own unused copies of the very same files (ols ×3, pandas_panel ×5 across programming and python.myst). Tracked in #337.

3. Cross-series file reuse is exactly one trio today (the pandas_panel CSVs), but the underlying sources recur everywhere — FRED alone is consumed three different ways in 10 lectures, plus World Bank, SCF, Maddison, OECD, Forbes, Yahoo. The consolidation upside in motivation 7 is real; it just materializes at the source level rather than the file level.

4. Provenance is the second axis the convention needs, alongside static/dynamic. Of the 34 files: 7 constructed with a committed builder (the hansen_singleton make_data.py + README pattern is the gold standard), 5 constructed but the pipeline was never committed (documented in prose only — unreproducible today), 7 verbatim third-party files (citation is the provenance; note rehosting these raises a licensing question the QEP should answer), and 14 author-assembled with no pipeline at all. Suggests three documented classes: verbatim (metadata only), constructed (builder required in the data repo), dynamic snapshot (builder + refresh cadence).

5. On live APIs: only 3 of the 11 lectures use the API as pedagogy (programming's pandas lecture teaches wbgapi/yfinance access; intro's business_cycle and inequality teach series discovery). The other 8 just need a series and are snapshot-ready. Our own history shows the migration pressure: hansen_singleton went datareader → direct downloads → fully vendored with maintenance scripts in two consecutive PRs (QuantEcon/lecture-python.myst#925, QuantEcon/lecture-python.myst#926), programming evicted pandas_datareader twice (QuantEcon/lecture-python-programming#341, QuantEcon/lecture-python-programming#133), and the live business_cycle lecture still needed a manual data refresh (QuantEcon/lecture-python-intro#770). Proposed rule for the manual: live APIs are for teaching data access, not for getting data — and every live-API lecture gets a snapshot twin built by a scheduled, validated refresh pipeline (the dynamic/ + scripts/ split in QuantEcon/data already prototypes this; it just has no automation — the repo has no .github/ at all).

Process

This supports the QEP direction, with the pilot (#338) as the empirical step in between: migrate one dataset per hosting pattern (local-path → cross-series → LFS/external → dynamic twin), fold what we learn into a draft styleguide/datasets.md, then write the QEP recording a convention that has already worked.

The full registry behind this digest — every file with contents, consuming lectures, hosting pattern, and provenance class, plus the URL-spelling inventory, orphan list, API classification, and the draft dynamic-data framework — is browsable here: https://claude.ai/code/artifact/87174334-f9e0-4b5c-b4df-0739a5abcdc2

The draft manual page is now up as the working design surface: QuantEcon/QuantEcon.manual#108 — styleguide/datasets.md (routing rule, URL form, provenance classes, live-API rule, dynamic refresh framework) plus the data-{scope} addition to styleguide/repos.md. Unsettled decisions are marked (open) inline; suggest we push design changes there as this thread converges, and merge it as the QEP's implementation.

@mmcky

Thanks @mmcky ! pls ping me if you need my input.

Complete Phase 1 (identity): repo renamed to data-lectures

Add PLAN.md, AGENTS.md and rewrite README (Phase 0 scaffolding) (#9)

Read the SCF sample from high_dim_data main (#793)

Read the SCF sample from high_dim_data main (#50)

Read the SCF sample from high_dim_data main (#48)

Read the SCF sample from high_dim_data main (#222)

Repoint ols data reads to this repo's own copies (#966)

Retire the last references to the retired lecture-python repo (#968)

Retire the last references to the retired lecture-python repo (#576)

Status — 2026-07-16

A working session against this project (plan: QuantEcon/workspace-lectures#14). Short version: both live risks in #337 are closed and published, the data repo has the layout the convention describes, and the P1 pilot is built and in review with findings already fed back to QuantEcon/QuantEcon.manual#108.

Landed

Lecture data: fix live hosting risks and bring high_dim_data into shape #337 risk 1 — the SCF branch pin: closed. The file was never stranded; it had been sitting in a mergeable PR ([new_dataset] SCF_plus_mini_no_weights high_dim_data#6) since 2023-04-28. Merged, all consumers repointed, branch gone. Verified live: the notebook a reader downloads from intro.quantecon.org now fetches its data and runs.
Lecture data: fix live hosting risks and bring high_dim_data into shape #337 risk 2 — all 8 legacy QuantEcon/lecture-python references: retired. Repoint ols data reads to this repo's own copies lecture-python.myst#966 (ols ×3), Retire the last references to the retired lecture-python repo lecture-python.myst#968 (pandas_panel ×2), Retire the last references to the retired lecture-python repo lecture-python-programming#576 (pandas_panel ×3). Every one gated on a sha256 byte-compare, and both series published — python.quantecon.org and python-programming.quantecon.org now serve zero legacy URLs, in the pages and the downloadable notebooks.
data-lectures onboarded to the lectures workspace (QuantEcon/workspace-lectures#15) so data-repo PRs and lecture-repo PRs can be coordinated from one place.

In review

Flatten the consumer-keyed tree into the published layout data-lectures#10 — the flat published tree (PLAN Phase 2). Done now precisely because zero lectures reference the repo, verified by org-wide search and not just the audit: every move is free today and a breaking change the moment the first repoint merges.
P1 pilot: add lingcod_msy_recovery.csv with its manifest data-lectures#12 → P1 pilot: read lingcod data from data-lectures lecture-python-intro#792 — pilot P1 (lingcod), the first dataset in the published tree and the first manifest.

What the pilot actually taught us

Filed in full on QuantEcon/QuantEcon.manual#108. The two that most affect this design:

The URL form needs to be chosen, not described. I wrote nine repoints today and drifted across three forms without noticing; Copilot caught it. They are not interchangeable — raw.githubusercontent.com returns pointer text rather than data for LFS-tracked files, so it fails as a confusing parse error instead of an honest 404. That is live today on high_dim_data. If "storage is invisible to consumers" is the goal, the styleguide has to name one form and say why.

Integrity is two questions, not one. P1's file is a byte-perfect copy of what the lecture consumed and has provenance that cannot be re-derived at all — both true simultaneously. One verified flag has to overstate or understate it, so the manifest now separates migration from upstream, and unverifiable with a reason is a legitimate value.

Ahead

P2 (pandas_panel) is next and is unblocked in a way it wasn't this morning — its trio now reads from own-repo copies rather than a dormant repo, so the migration starts from a clean base. Two repo-lifecycle threads came out of the sweep and are tracked in the workspace rather than here: QuantEcon/workspace-lectures#17 (rename lecture-python → lecture-python.rst and archive it — now unblocked) and QuantEcon/workspace-lectures#18 (the topic-based series: six live sites, 144 lectures, none linked from quantecon.org).

@mmcky

Thanks @mmcky , excited to see this come together

Flatten the consumer-keyed tree into the published layout (#10)

P1 pilot: add lingcod_msy_recovery.csv with its manifest

P1 pilot: add lingcod_msy_recovery.csv with its manifest (#12)

P1 pilot: read lingcod data from data-lectures (#792)

Add count and waiting-time datasets for a distribution-fitting lecture

Status — 2026-08-07: the proposal is built, running, and has taught us five rules it did not anticipate

The canonical repository proposed here exists as QuantEcon/data-lectures, it is serving data to four lecture repos today, and the convention has survived enough real repoints to have been revised by them. This is a progress report for people who have not followed the day-to-day, and it supersedes the numbers in the 2026-07-16 status comment.

Where it stands, measured

Numbers below are from a strict audit re-run today against current main of all eight Python-family repos (scripts/build_audit.py scan --strict, exit 0, zero problems in all four problem categories).

	2026-07-15 (the audit in this thread)	2026-08-07
Distinct static datasets in scope	34	41
Migrated and repointed	0	13
Committed data files across the repos	50	54
Committed orphans (files no lecture reads)	20	32
Distinct GitHub raw-URL spellings in use	6	5
References to the dormant `lecture-python` repo	8	0
Lectures fetching live API data	11	22

data-lectures holds 18 sidecar manifests covering 18 of the 19 datasets in the published tree — the exception is business_cycle_data.csv, the never-adopted dynamic snapshot, which is also the only file that would exercise class: dynamic-snapshot and cadence. The lifecycle tracker migration.yml records 13 repointed + 5 landed and zero pending or final. Four repos read from data-lectures today: lecture-python-intro, lecture-wasm, lecture-python.myst, lecture-python-programming.

Two pieces of machinery back this up. A required status check (consumed-files, from consumed-file-check.yml) hashes every manifested dataset that has consumers and fails the PR if the committed bytes do not match integrity.sha256 — live since 2026-07-17, alongside a protect-main ruleset. And a generated audit dashboard rebuilds on every push to main, weekly, and on relevant PRs; it scans all eight repos' main branches, classifies every data reference, and runs eight consistency checks between migration.yml, the manifests and what the lectures actually read. When strict fails it opens or updates a single audit-drift issue rather than going red unnoticed.

Q1–Q4 from the opening post are answered: name = data-lectures (settled 2026-07-16); layout = flat published lectures/ tree; pinning = main, with corrections fixed in place and new vintages getting a new filename; URL form = answered, but not with a single form — see rule 5 below.

The restructure: from hosting-pattern pilots to tracks by consuming series

The original plan decomposed the work by hosting pattern — P1 local-path, P2 cross-series, P3 external-repo-with-LFS, P4 dynamic snapshot. That was the right shape for validating the convention and it did its job: P1 and P2 both completed on 2026-07-17 and settled the metadata questions. It is the wrong shape for executing the remaining migration, and the work has been re-cut into tracks by consuming series.

The reason is that risk in this programme is concentrated in the repoint, not in the file move, and repoint risk is a property of who reads the file. Two repos reading the same dataset must move together or main is broken in between; a repo that publishes on a tag needs a different sequence from one that publishes on push; a lecture that reads by cell position needs its file and its lecture moved in the same change. None of those constraints are visible when you sort by hosting pattern, and all of them are obvious when you sort by consumer. Sorting by consumer also lets tracks run independently and in parallel, which the pattern-based waves could not.

Track	Scope	Remaining	Blocker
A	`lecture-python-intro` + `lecture-wasm` — paired, always repointed together	14	nothing to start. Includes the six `high_dim_data` files; `usa-gini-nwealth-tincome-lincome.csv` must follow the SCF files because it is built from one of them
B	`lecture-python.myst` — the `.dta` set, Hansen–Singleton, `NEWQDATA.csv`	7	none
C	`lecture-python-advanced.myst`	6	none (builder recovery is in-wave work, not a gate)
D	`lecture-python-programming` — `test_pwt.csv`	1	none; a single-PR track
E	dynamic / live-API — the UNRATE snapshot twin, then the incidental API lectures	new files, no existing static file moves	QuantEcon/data-lectures#14 (builder architecture and three schema decisions) and QuantEcon/data-lectures#26 (the fetch layer)
X	orphan sweep — dead committed files	32 across 6 repos	each repo's own repoints landing first
Y	infra / cutover — DNS, custom domain, interim→final URL sweep, QEP	—	an external ownership answer, see below

A+B+C+D = 28, which is the 41 in scope less the 13 already done. lecture-dp, lecture-jax and continuous_time_mcs are not data consumers at all — they appear only in Track X.

The operating rules the sweep produced

This thread's pilot plan described the bulk sweep as "mechanical once the convention is proven". It was not. Five rules came out of it, three of them learned twice, and they are the most transferable output of the last month.

Rule 1 — never delete a file a sibling repo reads. lecture-wasm mirrors lecture-python-intro's sources and fetches intro's committed blobs by URL. Deleting intro's copy in the repoint PR 404s the wasm build immediately. The sibling repoints first or together; the deletion goes in the same set.

Rule 2 — a coordinated change must land together, not merely be opened together. There is no green state for a partially-repointed dataset: the audit fails a record marked pending/landed while any consumer already reads data-lectures, and fails repointed/final while any consumer still does not. On 2026-08-06 QuantEcon/lecture-wasm#53 merged at 06:39Z, ahead of QuantEcon/lecture-python-intro#824 at 07:26Z, and left main failing on both files until the second landed. One branch name across all repos, PRs opened together, lecture repoints merged first, then the tracker flip.

Rule 3 — the published site lags main, so repoint, publish, then delete. Seven of the eight lecture repos publish on a publish tag — publish* for six of them, publish-* for lecture-dp — not on push to main; only lecture-wasm self-heals on merge. Merging a repoint therefore does not refresh the live site: already-published notebooks keep the old URL and 404 if the same PR deleted the file. After QuantEcon/lecture-python-intro#823 merged, the notebook served at intro.quantecon.org still fetched a mpd2020.xlsx that had just been deleted, and the window stayed open until a publish tag was cut. Rendered HTML is fine — figures are baked at build time; the breakage is confined to the downloadable notebook, the Colab link and {download} targets.

Two more, later:

Rule 4 — a migration moves bytes; it does not update them. Land the copy the lectures already consume, validated byte-identical — that is what makes a repoint provably unable to change a figure. Where our copy differs from today's upstream, migrate unchanged, record the difference in the manifest and in the register at QuantEcon/data-lectures#39, and review after the migration. This produced a new manifest status, diverged, with fields recording what differs and how the kind of difference was established — because "upstream moved" and "we edited it" look identical and need opposite responses. One dataset is in that state today: mpd2020.xlsx, where three header labels on the Regional data sheet were edited locally, and those edits are load-bearing for the lecture that reads it.

Rule 5 — in lecture-wasm the URL form is part of the contract. wasm executes code cells in the reader's browser, so every pd.read_*(url) is a cross-origin fetch, CORS-checked on every redirect hop. The github.com/<org>/<repo>/raw/… form is a 302 whose response carries an empty access-control-allow-origin, so the browser rejects it before following. This cuts against the draft styleguide, which carries a single canonical form (data.quantecon.org/lectures/…) plus a table of forms to avoid built around the LFS raw/media trap: the correct rule is two forms keyed by the consumer runtime — CPython consumers may use any form that resolves; browser consumers must use a direct host. We learned this by breaking it: a repoint normalised wasm's URLs onto intro's form and took out two wasm lectures, caught by an independent validation pass and repaired in QuantEcon/lecture-wasm#54. The audit now fails strict on it, but that is a post-merge net, not a gate — it scans each repo's main, so a violation turns the dashboard red at the next run rather than blocking the PR.

A related host trap worth recording, measured today: media.githubusercontent.com serves only LFS objects and 404s on plain-git files, while raw.githubusercontent.com serves the plain-git file. Both send access-control-allow-origin: *, so this is host routing, not CORS. Because the published tree is deliberately 100% plain git, the six high_dim_data datasets — which are LFS-tracked today and therefore read via the media host by twelve of their fourteen consuming reads — must change host as well as org and repo. A mechanical org/repo swap that preserves the host 404s on every one. The plain-git decision does not dissolve the raw-vs-media trap for the repoint; it inverts it.

The blind spot: prose references

The audit only sees data URLs inside executed code cells. It does not see {download} roles, prose links, _static/ builder notebooks, or _toc.yml. Ten such references to files the migration will delete are known and tracked in QuantEcon/data-lectures#42, and they are exactly the class of thing rule 3 breaks quietly. It also cannot see rule 3 at all, since it reads main and never the published site.

What is genuinely open, and is governance rather than engineering

Two items are not blocked on anyone writing code.

The domain. data.quantecon.org is a bare A record pointing at an AWS load balancer that returns 503 on every hostname and now presents a certificate for a third-party domain. The service behind it is gone; the balancer is still resolving. The fix is the same one already made three times for www, python and intro — a CNAME to quantecon.github.io — but it needs someone to confirm who owns that AWS account and whether anything else still points at the IP. That is escalated at #353. Worth being explicit that this is tidiness and stability, not a functional blocker: the interim GitHub Pages host already returns access-control-allow-origin: *, so the CORS requirement in the opening post is met today. The domain gates only the interim→final URL sweep, which is one coordinated sweep rather than per-dataset work.

The licensing inventory. The draft styleguide's checklist says confirm the licence permits redistribution before adding a dataset. That is right for genuinely new data and wrong for inherited data, and holding the migration on it would have stalled everything. The settled policy is record and track: anything the lecture repos already serve publicly migrates with its licence recorded exactly as found, including redistribution: restricted and name: null, with the honest state visible in the generated catalog from day one. Of the 18 manifests, 17 are permitted and one is restricted — countries.csv, © WorldData.info, tracked at QuantEcon/workspace-lectures#20. Eight carry license.name: null, each with a written reason. The gate this moves the question to is promotion of data.quantecon.org as a public open-data host — that is the point at which the restricted and unnamed entries need an answer, and it is a decision for the org, not for the migration.

Tracks A–D can run in any order. The remaining process debt is that every validated decision since 2026-07-15 lives in issue comments rather than in the styleguide — QuantEcon/QuantEcon.manual#108 has not been touched since 2026-07-16 and contradicts current practice in several places, most importantly on URL form and on licensing. Folding those back in is the precondition for graduating this to a QEP, which is where this thread said it should end up.

Correction and status, 2026-08-10

Five figures in my 2026-08-06 status comment have moved, and one of them is the exact error this programme has now made twice — so it is worth naming rather than quietly restating.

The high_dim_data read count. That comment said the six datasets are "read via the media host by twelve of their fourteen consuming reads". Twelve of fourteen is the lecture-python-intro + lecture-wasm subset mistaken for the total. The real scope of the fold's repoint is 21 reads across three repos — 17 on the media host, 4 on the github.com/*/raw/ redirect form: intro 5+2, wasm 7+0, lecture-intro.zh-cn 5+2. Org-wide it is 34, adding test-actions-lecture-intro (7) and lecture-python-intro.notebooks (6). The 12 figure is now flagged in PLAN.md by name, because it has been mistaken for the total once before.

Other numbers. Migrated 13 → 18; Track A remaining 14 → 9; committed orphans 32 → 26; "A+B+C+D = 28 less the 13" → 23 less 18. All re-derived from the generated audit.json today rather than restated.

Five repoint rules → six. Rule 6 (the media host is LFS-only, so a fold changes host as well as org and repo) landed 2026-08-07 in QuantEcon/data-lectures#50.

Rule 5 "gained a check" → three checks now. QuantEcon/data-lectures#48 covers rule 5; QuantEcon/data-lectures#55 (merged 2026-08-10) added rule 6 and a ref/path resolvability assertion, deliberately as two independent checks — a media URL parses to a byte-identical (ref, path) as the raw URL beside it, so neither can stand in for the other.

DNS. That comment described data.quantecon.org as "a bare A record pointing at an AWS load balancer that returns 503". Measured today: the name is NXDOMAIN, with the authoritative flag set, at quantecon.org's own nameserver — the record is gone — and the repo's Pages cname is null. So Track Y's blocker is no longer an external-ownership question; it is two actions QuantEcon controls: create the record, then set the custom domain. Mirrored onto #353.

One new structural finding for the convention itself. build_audit.py's URL classifier recognises six GitHub-host regexes and neither quantecon.github.io nor data.quantecon.org. So a consumer reading from the canonical host classifies as external-web, migrated evaluates false, and the migration tracker's terminal status — "every consumer reads data.quantecon.org" — is a state the audit is structurally guaranteed to report as broken. Both of the new host assertions also lapse at that moment. Worth knowing before the convention specifies a canonical URL form: the tooling has to learn the canonical host before the cutover, not after.

The storage question that ran underneath all of this is now written up in full at QuantEcon/data-lectures#58 — the platform limits, why LFS stays out of a published tree, the ladder for a file above 100 MiB, and the finding that the manifest schema records nothing about storage (and the recommendation that it should not start).

This issue is the `infrastructure` programme's front door on the projects dashboard

Recording it here rather than only in the registry, so anyone arriving at this thread knows what it is being used for.

QuantEcon's portfolio dashboard — https://quantecon.github.io/status-projects/ — groups registered projects into eleven programmes, and each programme may declare a front door: the issue where that programme's projects are coordinated between themselves. This issue is that front door for infrastructure, alongside the datasets work it proposes.

The programme's projects appear at https://quantecon.github.io/status-projects/#programme-infrastructure, with stage, priority, owner and observed health for each. Nothing on this thread is read for those facts — a front door is a destination for a reader, not a source the collector parses. That is the tracker issue's job, one per project.

One thing worth flagging on this thread rather than deciding quietly. On 2026-09-08 the dashboard's owner ruled that the two genuine programme trackers should carry the org's native Project issue type, and deliberately held this issue and #332 back from it. The reason: both are titled Proposal, and this one has three sub-issues while #332 has none — typing a proposal as a project asserts something that may not be true, and the org has a Decision type that fits a proposal better.

That leaves a real question, which is open at QuantEcon/status-projects#113: should a proposal serve as a programme's front door at all, or does a programme need a coordinating tracker distinct from the proposal that started it? If this thread graduates to a QEP as its opening line suggests, the question answers itself and the front door should move. If not, it is worth settling on its own terms.

No action is being asked of this thread; it is recorded so the decision is visible where it applies.

Add us_household_net_worth_2022.csv: SCF 2022 household net worth wit…

Proposal: a single canonical QuantEcon datasets repository, with a documented convention #336

Description

Summary

Motivation

Proposal

Alternatives considered

Rollout

Smaller issues to open regardless of the outcome

Open questions for the team

Recommendation on process

Activity

mmcky commented on Jul 15, 2026

What exists today

Q1 (name): data-lectures

Q4 (URL form): a trap worth knowing about

Which is why I'd propose data.quantecon.org/lectures

Q2 (layout): flat

Q3 (pinning): main, with git history as the version record

Process

mmcky commented on Jul 15, 2026

Headline numbers

Findings that sharpen the proposal

Process

mmcky commented on Jul 15, 2026

jstac commented on Jul 15, 2026

mmcky commented on Jul 16, 2026

Status — 2026-07-16

Landed

In review

What the pilot actually taught us

Ahead

jstac commented on Jul 16, 2026

mmcky commented on Aug 6, 2026

Status — 2026-08-07: the proposal is built, running, and has taught us five rules it did not anticipate

Where it stands, measured

The restructure: from hosting-pattern pilots to tracks by consuming series

The operating rules the sweep produced

The blind spot: prose references

What is genuinely open, and is governance rather than engineering

Next

mmcky commented on Aug 10, 2026

Correction and status, 2026-08-10

mmcky commented on Sep 8, 2026

This issue is the infrastructure programme's front door on the projects dashboard

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Q1 (name): `data-lectures`

Which is why I'd propose `data.quantecon.org/lectures`

Q3 (pinning): `main`, with git history as the version record

This issue is the `infrastructure` programme's front door on the projects dashboard