Repository navigation
Do the business_cycle .md dumps belong in the published lectures/ tree? #13
Description
Follow-up from PR #10 (now merged). Flagging a layout question that the flatten deliberately left open.
The business_cycle.py builder writes three files into lectures/, and everything under lectures/ is servable at data.quantecon.org/lectures/<filename>:
| File | What it is | Fetched by a lecture? |
|---|---|---|
business_cycle_data.csv |
the actual dataset — annual GDP growth for 5 economies | yes |
business_cycle_metadata.md |
World Bank series metadata (source, licence CC BY-4.0, periodicity) | no — provenance |
business_cycle_info.md |
the string dump of wb.series.info(q='GDP growth'), a fuzzy search-results listing |
no — provenance byproduct |
The question is whether the two .md files are published datasets or provenance byproducts of the builder. business_cycle_info.md is the weakest case: it is a search-results listing, not a dataset any consumer would fetch, yet it becomes a public URL at data.quantecon.org/lectures/business_cycle_info.md.
business_cycle_metadata.md is more defensible — it is the record proving the CC BY-4.0 licence, and PLAN Phase 6 already treats it as the model for what a manifest should capture. That role points toward it becoming a manifest sidecar (business_cycle_data.csv.yml) rather than a served .md, once the schema from #12 lands.
Options
- Keep both in
lectures/— the AGENTS.md repo map already names them ("business_cycle's upstream metadata dumps"), so this is a conscious choice; just make the justification explicit in AGENTS.md rather than leaving it implied. - Move both out of the published tree — have the builder write them somewhere unserved (e.g. alongside
scripts/, or aprovenance/dir) so only the.csvis a public URL. - Split them — fold
business_cycle_metadata.md's content into the Phase 6 manifest sidecar and dropbusiness_cycle_info.mdfrom the published namespace entirely.
No consumer references either file today (org-wide search returned zero at PR #10 time), so like the flatten itself this is free to settle now and gets more expensive after the first repoint (Phase 8).
Part of #8
Activity
mmcky commentedon Aug 6, 2026
Recommendation: option 3 (split them), and it has got more expensive since this was opened
Re-checked against
maintoday (2026-08-06). The situation has changed in one way that matters: when this issue was written the flatten had landed but Pages had not, so "becomes a public URL" was a future tense. It is now present tense — all three files are live and served:quantecon.github.io/data-lectures/lectures/business_cycle_data.csvquantecon.github.io/data-lectures/lectures/business_cycle_metadata.mdquantecon.github.io/data-lectures/lectures/business_cycle_info.mdSo
business_cycle_info.md— awb.series.info(q='GDP growth')fuzzy-search dump — is already a published artifact of this repo, sitting in the same namespace and indistinguishable from a dataset to anyone browsing/lectures/.The good news is that removal is still cheap. A fresh strict audit confirms zero consumers for all three files — no lecture in any of the 8 scanned repos reads any of them, and none has a manifest. Nothing breaks.
Why option 3 rather than 1 or 2
business_cycle_info.mdshould leave the published namespace. It is a search-results listing, not a dataset. Keeping it (option 1) means the repo's public tree contains an artifact that fails its own rules — no class, no manifest, no consumer, no licence — which undercutsAGENTS.md's "every dataset needs a class and a manifest" the moment anyone looks. It is genuinely useful as provenance, so it should live where the builder can write it without publishing it.business_cycle_metadata.mdshould become manifest content, not a served file. It records the World Bank CC BY-4.0 terms and the series metadata, which is exactly what thesourceandlicenseblocks of a sidecar manifest exist to carry.PLAN.mdPhase 6 already calls it "the model for what a manifest should capture" — so the natural end state is that its content isbusiness_cycle_data.csv.yml, not a.mdsitting beside it. Keeping both means one fact recorded in two places that can drift.Option 2 (move both out of the tree) is defensible and is the cheap version of this, but it leaves the licence record in a free-text
.mdthat nothing validates, when the schema already has typed fields for it.Suggested shape
Have
scripts/business_cycle.pywrite onlybusiness_cycle_data.csvintolectures/, and write its two provenance dumps to an unserved location —provenance/at the repo root is the obvious choice, sitting outsidelectures/exactly asscripts/does. Fold the World Bank source, licence and periodicity frombusiness_cycle_metadata.mdinto a newlectures/business_cycle_data.csv.yml, withcadencedeclared since this is the repo's one dynamic snapshot.When
This lands naturally inside the P4 / dynamic-snapshot work rather than as a standalone PR, because
business_cycle.pyis being retrofitted to the four-stage contract there anyway (PLAN.mdPhase 5, and #14), and writing the manifest is the step that forces thecadenceand schema decisions. Doing it as part of that keeps one builder rewrite instead of two.The one argument for doing it now instead: every week it stays, the chance rises that something starts linking
business_cycle_info.md, and the removal stops being free. It is a two-file move with zero consumers — happy to split it out as its own small PR if you would rather bank it immediately.Part of #8. Related: #14 (the builder retrofit), and the
business_cyclesnapshot-adoption question inPLAN.mdPhase 8 P4.P4 groundwork: manifest, four-stage builder and provenance/ for busin…
mmcky commentedon Sep 1, 2026
Settled as option 3 in #109, merged 2026-09-01.
business_cycle_info.mdandbusiness_cycle_metadata.mdleftlectures/for a new unservedprovenance/directory (the Pages job assemblessite/,lectures/andaudit.jsononly), with a README saying what they are and why they exist. The World Bank source, CC BY-4.0 licence and annual cadence from the metadata dump are now typed fields inlectures/business_cycle_data.csv.yml— the manifest this issue predicted would absorb them — and the dump is kept as the evidence those fields cite, regenerated on every builder run. Zero readers of either file org-wide at removal, by code search and the audit. The wider P4 work this rode in with (four-stage builder, the diverged-by-design integrity record) is on #109 and #39.