Repository navigation
Lecture data: fix live hosting risks and bring high_dim_data into shape #337
Description
Part of #336. A full audit of dataset usage across the 7 Python lecture repos (2026-07-15, each repo's main) surfaced two live breakage risks plus a backlog of shape-up work on the data repos. This issue documents them so the fixes land as part of the consolidation project rather than ad hoc.
Live risk 1 — a lecture reads a file that exists only on a feature branch
lecture-python-intro/lectures/mle.md (line 93) reads SCF_plus_mini_no_weights.csv from the update_scf_noweights branch of high_dim_data. The audit confirmed this file does not exist on main at all — the feature branch is its only home. Deleting or rebasing that branch is an instant 404 for the lecture, so this is sharper than the "latent breakage" noted in #336.
- Land
SCF_plus_mini_no_weights.csv(and its generating pipeline) onhigh_dim_datamain - Repoint
mle.mdto amain-based URL (or straight to the canonical repo if Proposal: a single canonical QuantEcon datasets repository, with a documented convention #336 has settled by then) - Delete the
update_scf_noweightsbranch once nothing references it
Live risk 2 — eight URLs depend on the dormant pre-MyST repo
QuantEcon/lecture-python (the old Sphinx repo, last pushed January 2022, not archived) still serves data to two current series. Deleting it today would break 3 lectures. In every case the consuming repo already commits its own copy of the same file — the fetch just points at the wrong repo.
Correction, 2026-07-16. This originally read "Archiving or deleting it today would break 3 lectures". Archiving turned out to be harmless: archived repositories keep serving raw content indefinitely, verified live against
QuantEcon/lecture-python-website(archived, Pages stillbuiltand serving) and again after this repo was archived. What actually breaks consumers is deletion, or reuse of the freed name after a rename — the latter tested on the twin, wherelecture-python-programmingwas renamed to.rstand the MyST repo took the vacated name, after which old URLs resolve against the new occupant instead of the archived content.
| Consuming repo | Lecture | Refs | Currently fetched from |
|---|---|---|---|
| lecture-python-programming | pandas_panel | 3 (realwage, countries, employ) | lecture-python master |
| lecture-python.myst | pandas_panel | 2 (realwage, countries; employ already repointed) | lecture-python master |
| lecture-python.myst | ols | 3 (maketable1/2/4 in the body; the solutions already read the own-repo copies) | lecture-python master via blob/…?raw=true |
- Repoint all 8 references (interim: own-repo copies; final: the canonical data repo per Proposal: a single canonical QuantEcon datasets repository, with a documented convention #336). Interim repointing is churn-tolerant — moving twice beats breaking when someone archives the old repo
- Archive
QuantEcon/lecture-python— done 2026-07-16: renamed tolecture-python.rst, description corrected, README warning box added, then archived. Details in QuantEcon/workspace-lectures#17. The "once nothing references it" precondition proved unnecessary — archived repos keep serving raw content, so the 15 references that remain (lecture-python.zh-cn9,lecture-stats6) still resolve. Re-verified after archiving: all three URL forms in use return 200.
Shape-up backlog — high_dim_data
- Rewrite the README — the repo is de-facto general lecture data, not "High Dimensional Economics" (see Proposal: a single canonical QuantEcon datasets repository, with a documented convention #336)
- Fix
.gitattributes: currently*.csv filter=lfstracks every CSV regardless of size; switch to per-path LFS (relates to the bandwidth concern in monitoring bandwidth usage high_dim_data#4 and the raw-vs-media URL trap documented in Proposal: a single canonical QuantEcon datasets repository, with a documented convention #336) - Execute the Proposal: a single canonical QuantEcon datasets repository, with a documented convention #336 outcome: fold into the canonical data repo (Migrate
high-dim-datato this repo data-lectures#2) and sweep the 6 consuming references in lecture-python-intro (inequality, heavy_tails, mle)
Related cleanup — orphaned data files in the lecture repos
The audit found 20 of 50 committed data files are dead weight: orphans with no reference, copies superseded by cross-repo fetches, or files shadowed by %%file cells that regenerate them at build time. Notably lecture-dp carries 10 unused files inherited from lecture-python.myst, and lecture-python-intro still commits two World Bank CSVs whose canonical copies moved to QuantEcon/data in Feb 2025.
- Sweep the orphans per repo (intro 3, programming 7, python.myst 5, dp 10 — full list available from the audit registry)
None of the checklist items above depend on the #336 design decisions except the two "final home" repoints — everything else can proceed now.
Activity
Read the SCF sample from high_dim_data main (#793)
Read the SCF sample from high_dim_data main (#50)
Read the SCF sample from high_dim_data main (#48)
Read the SCF sample from high_dim_data main (#222)
Repoint ols data reads to this repo's own copies (#966)
Retire the last references to the retired lecture-python repo (#968)
Use this repo's existing raw.githubusercontent URL form
Retire the last references to the retired lecture-python repo (#576)
mmcky commentedon Jul 16, 2026
Both live risks are closed — merged, published, and verified live
Worked through in a session tracked at QuantEcon/workspace-lectures#14.
Risk 1 — the SCF branch pin ✅
main— it was never stranded on an unreviewed branch. It had been sitting in [new_dataset] SCF_plus_mini_no_weights high_dim_data#6 since 2023-04-28 (@shlff),MERGEABLEand conflict-free the whole time: the branch was 5 behindmain, butmainonly touchedcross_section/while the branch only touchedSCF_plus/. Merged 2026-07-15.main-based URL — done, in four repos (see below).The one thing worth carrying forward: the pin had 4 consumers, not 1
This issue tracked
lecture-python-intro/lectures/mle.md. An org-wide search found the same pinned URL in four source repos:lecture-python-introlecture-intro.zh-cn— live Chinese translationlecture-wasmtest-lecture-python-intro— actions fixture, genuinely buildsmle.md(
lecture-python-intro.notebooksalso carried it but is generated — it self-healed on the next publish.)And the breakage arrived immediately, not later.
high_dim_datahas automatically delete head branches enabled, so merging #6 deletedupdate_scf_noweightson the spot — 404ing the URL all four repos were still using. All four were repointed and merged within the hour, andmleis verified working end-to-end: the notebook downloaded fromintro.quantecon.org/_notebooks/mle.ipynbfetches its data and runs (10,000-row sample, mean 6.5642).The lesson for the rest of this project: if a branch-pinned URL exists anywhere, land the repoints before merging anything that could remove the branch. The window between "merge" and "repoint" is not theoretical.
Bytes are provably unchanged — the LFS object on
maincarries the same OID the branch served (sha256:c3b2c78a…, 75902999 bytes).Risk 2 — all 8 legacy references retired ✅
The table in this issue was exactly right. All 8 are gone:
lecture-python.mystlecture-python.mystlecture-python-programmingQuantEcon/lecture-python— now unblocked; tracked in QuantEcon/workspace-lectures#17, which also proposes renaming it tolecture-python.rstfirst (it is unambiguously RST: 74.rstfiles undersource/rst/, built by Sphinx).Every repoint was gated on a sha256 byte-compare — own-repo copy vs legacy file vs what the new URL serves — so no lecture output moved. Both series are also published, so this is real for readers rather than just true on
main:python.quantecon.org/pandas_panel.htmlpython-programming.quantecon.org/pandas_panel.html#968 also fixed something adjacent:
lecture-python.myst/lectures/troubleshooting.mdwas telling readers to report problems with the live series by opening an issue on the retired repo — which still has issues enabled, so reports landed where nobody looks.Two things the audit did not cover
lecture-statsholds 7 more references (ols.md) and is not in the table above. It is being archived as part of the topic-based series (QuantEcon/workspace-lectures#18) — confirmed stale 2026-07-16 — so those references retire with the repo. No PR needed, and it does not block the archive.lecture-python.zh-cnholds 9 (ols.md5,pandas_panel.md3,mle.md1) and will not self-heal. Thelecture-python-programmingtranslations do — merging #576 firedsync-translations-{fa,fr,zh-cn}.ymlandQuantEcon/action-translationopened a PR in each target within seconds. Butlecture-python.mysthas no sync workflow, andlecture-python.zh-cnhas never been synced (no.translate/state). After the rename it would depend on GitHub's redirect permanently. Rolling the sync out tolecture-python.mystfixes it generally; that is the translation programme's call, not this issue's.Not started
The
high_dim_datashape-up backlog (README, per-path LFS, fold into the canonical repo) and the 20-file orphan sweep are untouched. The datasets-repo side of the orphan question did move: QuantEcon/data-lectures#10 drops the two World Bank CSVs andfig_3.odsfromdata-lecturesrather than promoting them into the published tree —fig_3.odswas verified to be a pure format twin offig_3.xlsx(identical single sheet, identical 34×6 frame), and the World Bank pair returns zero references org-wide.mmcky commentedon Aug 6, 2026
Closing status — both live hosting risks are resolved, one precondition remains
Both enumerated live risks are closed and have stayed closed. What is left in this issue is the
high_dim_datashape-up, and that work has moved: it is now Track A / P3 of the migration inQuantEcon/data-lectures, not a standalone tidy-up. The numbers below are measured, not estimated — a fresh strict run ofdata-lectures/scripts/build_audit.pyagainst clones of all eight Python-family repos on 2026-08-07 returnsstatic_files: 41, committed_files: 54, orphans: 32, legacy_refs: 0, migrated: 13, api_lectures: 22, url_forms: 5.Risk 1 — the branch-pinned SCF file
Closed.
mle.mdreadSCF_plus_mini_no_weights.csvfrom the branchupdate_scf_noweights, and the file did not exist onmain.QuantEcon/high_dim_data#6had sat mergeable since 2023-04-28; it merged 2026-07-15T23:57:12Z, which put the file onmainand made the branch pin removable. The pin had four consumers, not one —lecture-python-intro#793,lecture-wasm#50,lecture-intro.zh-cn#222andtest-lecture-python-intro#48, all merged between 00:16 and 00:21 on 2026-07-16, inside half an hour of the upstream merge;lecture-python-intro.notebooksself-healed on regeneration and now readsmain. The head branch was then deleted deliberately, once nothing referenced it —high_dim_datadoes not auto-delete head branches (delete_branch_on_merge: false, checked today), so the ordering was the repoints first, deletion after.Verified today:
high_dim_datahas exactly one branch,main. The branch is gone and cannot come back by accident, so this risk has no recurrence path.Risk 2 — the eight URLs into the dormant
lecture-pythonClosed. Repointed by
lecture-python.myst#966(the threeolsreads, onto that repo's own copies),lecture-python.myst#968andlecture-python-programming#576(thepandas_panelreads — five between them), every one gated on a sha256 byte-compare. Thepandas_paneltrio subsequently became the P2 migration pilot and now reads fromdata-lectures(lecture-python.myst#973,lecture-python-programming#578).Measured:
legacy_refs: 0across all eight repos. The dormant repo was renamed tolecture-python.rstand archived on 2026-07-16 (QuantEcon/workspace-lectures#17);gh apiconfirms today thatlecture-python.rstis archived and that thelecture-pythonname resolves through the rename redirect.Still outstanding and not ours to fix:
lecture-python.zh-cncarries its own copies of these references and will not self-heal —lecture-python.mysthas no sync workflow and zh-cn has no.translate/state. That belongs to the translation programme, not to this issue.Two facts this thread's readers should take away
An archived repo keeps serving
raw.githubusercontent.comand GitHub Pages indefinitely. Archiving is a safe terminal state — it freezes writes and signals dormancy without breaking a single consumer. What breaks consumers is deletion, and name reuse after a rename: a rename leaves a redirect, but the redirect dies the moment the old name is claimed by a new repo. This corrects the framing in this issue's original body, which treated archiving itself as the hazard.gh search codecannot find URLs. It tokenises on/and., so a search for a URL fragment returns a clean, confident zero. That is exactly how the reference counts in two tracking issues came to be wrong. The only reliable method is clone-and-grep, which is whatbuild_audit.pydoes.The orphan sweep — measured numbers
This issue estimated "20 of 50 committed files are dead weight". The measured figure across eight repos is 32 orphans of 54 committed data files:
The sweep is now Track X in
data-lectures/PLAN.md, sequenced behind each repo's repoints for two reasons discovered the hard way. Repoint rule 1: never delete a file a sibling repo reads by URL —lecture-wasmmirrors intro's sources, and most of its 11 orphans are mirror-copies that only become safe to delete once wasm readsdata-lecturesdirectly. Repoint rule 3: the published site lagsmain— seven of the eight lecture repos publish on apublish*tag rather than on push, so a merged deletion can 404 an already-published notebook until a publish is tagged. (lecture-wasmis the exception; it publishes on push tomain.)high_dim_datatodayVerified 2026-08-07: not archived, public, one branch (
main), last pushed 2026-07-15T23:57:23Z, three open issues (#4, #5, #7), all pre-dating this work. Its.gitattributesis still the blanket rule this issue asked to fix —*.dtaand*.csvbothfilter=lfs, which is why every data blob in the repo is an LFS pointer of ~134 bytes (SCF_plus/SCF_plus.dtafetches as 134 bytes over raw, standing in for a 103,934,093-byte object) and every consumer must read throughmedia.githubusercontent.com. All three shape-up boxes (README rewrite,.gitattributes, fold into the canonical repo) remain unticked.Fourteen live reads point at it today, seven in
lecture-python-introand seven inlecture-wasm:heavy_tails.md×4 in each,mle.md,inequality.md, and_static/lecture_specific/inequality/data.ipynbin each. Twelve use the media host; two — intro'smle.md:93andinequality.md:249— use thegithub.com/…/raw/redirect form.One operational note for whoever executes the fold, measured with curl today:
media.githubusercontent.comserves only LFS objects and returns 404 for plain-git files.raw.githubusercontent.com/QuantEcon/data-lectures/main/lectures/mpd2020.xlsx→ 200; the same path on the media host → 404. Both hosts sendaccess-control-allow-origin: *, so this is host routing, not CORS. Sincedata-lectureskeeps its published tree 100% plain git, the six datasets land there unLFS'd — so every one of those 14 reads must change host as well as org and repo. A mechanical org/repo swap that preserves the media host 404s on all twelve.The one remaining precondition before archiving
SCF_plus/generating_mini.md— the SCF builder that lives insidehigh_dim_data— reads its own input over the network from the repo it lives in:(line 37, verified on
maintoday). Archiving while that line stands does not break anything — an archived repo still serves raw — which is exactly the problem: it quietly re-introduces the dormant-repo dependency this issue drove to zero, in a file nobody will look at again. The builder must read its input from a localsources/path before the repo is archived. That is a general rule now (data-lectures/AGENTS.md): a builder reads fromsources/, never over the network from another QuantEcon repo.Recommendation
Close this issue. Both enumerated live risks are resolved, verified, and have no recurrence path; the correction about archiving is recorded above; the orphan sweep has been re-scoped, re-measured and re-sequenced under
data-lecturesTrack X, where its blockers actually live.The single remaining precondition — repoint
generating_mini.mdatsources/before archivinghigh_dim_data— is already carried indata-lectures/PLAN.mdPhase 3 and tracked as part of Track A / P3 (see QuantEcon/data-lectures#2 and QuantEcon/workspace-lectures#23). Keeping this issue open duplicates that tracking rather than adding to it.