Scaffolding: QuantEcon/data → data-lectures (canonical lecture-data repo) #8

Part of QuantEcon/meta#336 (design thread) — this issue documents the scaffolding and initial maintenance work to take this repo from its current state to the canonical data-lectures repository described in the draft convention (QuantEcon/QuantEcon.manual#108). The pilot (QuantEcon/meta#338) lands its migrations here, so the early items below are its prerequisites.

Checklist refreshed 2026-09-01 (previously 2026-08-06) — boxes reflect verified state of main, not intent. Closed 2026-09-01: the scaffolding is done and the static migration it was the prerequisite for completed 2026-08-18; the boxes still open below are Phase 5 automation, tracked in PLAN.md and #14 rather than here. PLAN.md remains the roadmap; this issue is the scaffolding subset of it.

Current state (audit, 2026-07-15 — superseded, kept for the record)

The repo holds 10 files for lecture-python-intro under a consumer-keyed layout (lecture-python-intro/static/, dynamic/, scripts/), has one manual refresh script (business_cycle.py), no .github/ directory at all (no CI, no automation, no scheduled refresh), no LFS, no per-dataset metadata, and is referenced by zero lectures — the sweep in #4 never happened.

Where it stands 2026-08-06: flat published tree, 10 datasets with manifests, all 10 repointed, .github/ with three workflows, Pages live and CORS-clean, branch protection with a required check. The audit dashboard is green against all 8 lecture repos: 41 static files, 35 orphans, 22 live-API lectures, 0 legacy refs.

Identity

• Rename QuantEcon/data → data-lectures once the name settles in Proposal: a single canonical QuantEcon datasets repository, with a documented convention meta#336 (GitHub redirects preserve existing URLs, so this is non-breaking and can go first) — done 2026-07-16, old raw URLs verified to 200 through the redirect
• Rewrite README.md: purpose, the routing rule, how to add a dataset, link to the manual page — done in Add PLAN.md, AGENTS.md and rewrite README (Phase 0 scaffolding) #9 (2026-07-16), together with PLAN.md (the phased roadmap for this issue) and AGENTS.md
• Add repo description and topics — done 2026-07-16

Layout

• Restructure from consumer-keyed (lecture-python-intro/...) to the flat published tree in the draft convention — no folder implies ownership by a series — done in Flatten the consumer-keyed tree into the published layout #10 (2026-07-16)
• Decide where non-published assets live (scripts/, per-dataset manifest.yml, tests) relative to the published tree — settled in Flatten the consumer-keyed tree into the published layout #10: scripts/ and manifest-schema.yml at the root, manifests as sidecars inside lectures/ named <full filename>.yml
• Generate an index/catalog page from the manifests — this doubles as the dataset registry — scripts/build_catalog.py → CATALOG.md, done in Add generated dataset catalog (CATALOG.md + build_catalog.py) #19

One layout question from the flatten is still open: whether business_cycle's two .md provenance dumps belong in the published tree at all — #13. Both are live public URLs today, so this got more expensive since it was raised.

Storage

• Per-path LFS via .gitattributes — large binaries only, small teaching files plain git (Setup Git LFS for large file support #1; avoid high_dim_data's blanket *.csv rule) — done 2026-08-10 (LFS: scope it to sources/**, and stop fetching it in CI #57), scoped to sources/** only; the published tree is plain git (Storage policy for large data files: plain git in lectures/, LFS only in sources/ — and the ladder above 100 MiB #58). Original note: note the upstream rule is *.csv and *.dta. Worth re-testing the premise: both SCF minis (31.3 MiB and 72.4 MiB) fit plain git under GitHub's 100 MiB limit, so LFS may only be needed for SCF_plus.dta (99.12 MiB), which has no lecture consumer
• Fold in high_dim_data content (Migrate high-dim-data to this repo #2; coordinates with Lecture data: fix live hosting risks and bring high_dim_data into shape meta#337 for the consuming-lecture repoints and the branch-only SCF file) — done 2026-08-11 (Fold in the six high_dim_data datasets (PR B1) #62, Land SCF_plus.dta in sources/, and gate sources/ on its recorded hashes (PR B2) #63, 28 reads repointed across four repos, flipped in Flip the six high_dim_data datasets to repointed #69). Original note: the branch-only file is resolved upstream, so the fold starts from a clean base

Publishing

• GitHub Pages deploy of the published tree, lfs: true at checkout so LFS objects publish as bytes rather than pointer files — done 2026-07-17 in Add the generated data-audit dashboard (gh-pages): audit + migration tracker #21; dashboard at /, published tree at /lectures/
• data.quantecon.org DNS + custom domain — deferred indefinitely 2026-08-12 (D11, PLAN-QELD-PACKAGE.md §2.2, recorded in Record D11: invest in qeld, defer data.quantecon.org indefinitely #76): the qeld package is the stable consumer interface instead of a branded host, so this item no longer gates anything — Phase 4 cutover: swap interim raw URLs → data.quantecon.org/lectures #15 is superseded by qeld adoption and closed, and Track Y (deferred): data.quantecon.org DNS + custom domain — retired in favor of qeld (D11); reopenable, NXDOMAIN as of 2026-08-10 #37 stays open as the deferred tracker. The external blocker also dissolved on its own: the stale A record was deleted and the name is NXDOMAIN (measured 2026-08-10), so reviving this is two actions QuantEcon controls whenever D11 is revisited
• Verify access-control-allow-origin: * on the served files (pyodide/JupyterLite requirement, [Plan/Idea] Supporting WASM based introductory series meta#143) — verified 2026-08-06 on the default Pages domain: quantecon.github.io/data-lectures/lectures/lingcod_msy_recovery.csv returns access-control-allow-origin: *. The requirement is satisfied today and does not wait on the custom domain; re-verify once DNS moves
• Monitor Pages soft limits (~1GB site, 100GB/month) — ongoing, nothing near the limit yet. Becomes live when the SCF files land

Automation (.github/)

Go-live guardrails, added ahead of the first repoint:

• Branch protection on main — PRs required, force-push and deletion blocked (protect-main ruleset, 2026-07-17)
• Consumed-file check as a required status check — .github/workflows/consumed-file-check.yml (2026-07-17)
• Audit dashboard workflow, strict mode — .github/workflows/audit-dashboard.yml (Build a generated data-audit dashboard (gh-pages) across the 8 Python-family repos #20, closed 2026-08-06), with a drift-alarm inbox added in Give the drift alarm an inbox — assign a failure issue to mmcky #29

Remaining:

• PR validation: manifest schema check + per-dataset validate stage on every PR touching data — blocked on Builder architecture: four-stage template, traceback-clean failures, notebook policy (Phase 5) #14's three schema decisions (column patterns, known_nulls exact-vs-ceiling, dtype vocabulary). The dtype vocabulary has already drifted across the ten manifests, so this is no longer cost-free to defer
• Scheduled refresh workflow for dynamic datasets — runs the builder, lands the result as a PR with a diff summary (first consumer: the UNRATE pilot, P4 in PLAN: pilot migration — one dataset per hosting pattern to validate the datasets design meta#338)
• Weekly "sources alive" canary: fetch + validate, no commit, opens an issue on failure
• Retrofit scripts/business_cycle.py to the four-stage contract — it still has no validate stage
• Package the refresh job as a reusable workflow (quantecon/actions) once it stabilizes

The fetch-layer question these builders depend on is #26 (pandas_datareader is maintained again).

Metadata backfill for existing holdings

• Per-dataset manifest schema — manifest-schema.yml, revised by P1 (P1 pilot: add lingcod_msy_recovery.csv with its manifest #12) and exercised by nine more datasets since
• [~] Per-dataset manifest for the existing files — 10 of 19 done. The ten migrated datasets all have sidecars; the 8 static intro files in the published tree (mpd2020.xlsx, longprices.xls, chapter_3.xlsx, assignat.xlsx, dette.xlsx, fig_3.xlsx, caron.npy, nom_balances.npy) still have none, and neither does business_cycle_data.csv. This is the next tranche of work
• Keep-or-drop for the three no-consumer files — dropped in Flatten the consumer-keyed tree into the published layout #10 (two World Bank CSVs, fig_3.ods), recoverable from history
• License check per file before the repo is promoted as the canonical public home — policy settled 2026-08-06: licensing does not gate migration. Data already served publicly by the lectures migrates with its status recorded in the manifest; anything needing further thought is tracked for review before data.quantecon.org is promoted as a public open-data host. See the inventory issue and QuantEcon/workspace-lectures#20

Adoption (the step that stalled in Feb 2025)

• [~] Repoint the consuming lectures as datasets land here (Add data and scripts #4's unticked box) — 10 of 41 done, all repointed in migration.yml. The remaining 31 are waved in PLAN.md
• Remove lecture repos' duplicate copies as each repoint merges — 24 orphans across 6 repos as of the 2026-08-31 audit (down from 35); every repoint has landed, so this is now Track X, tracked at QuantEcon/workspace-lectures#57

Two sequencing rules learned since and now recorded in PLAN.md: a dataset a sibling repo reads (every lecture-wasm case) must have that sibling repointed before the owning repo's copy is deleted, or the sibling 404s; and because the strict audit has no green state for a partially-repointed dataset, all consumers of one dataset must be repointed together. Sixteen of the thirty-one remaining datasets are multi-consumer, every one of them lecture-python-intro + lecture-wasm.