Skip to content

VALIDATION: independent review of the 2026-09-07 schema-decisions, rename and manifest-driven validator work #127

Closed
Closed
VALIDATION: independent review of the 2026-09-07 schema-decisions, rename and manifest-driven validator work#127

Description

@mmcky
mmcky
opened on Sep 7, 2026

Everything below was verified during the work by the same agent that did it, in one session on 2026-09-06/07: the four schema and naming decisions (#120, #121, #122, #113), the rename of business_cycle_data.csv, the shared manifest-driven validator and its PR check (#119), the close of #14, and the tracker and dashboard updates around them. Model for this issue: #45 and its validation run (#46 was found in the margin, not on the list).

Bias to test for. The session's verification monoculture is the validator it wrote: scripts/validate_datasets.py over builders/_validate.py, run locally through uv run under pandas 2.3.3 and 3.0.5, plus a scratchpad mutation script that was never committed. Every "44 of 44 pass" claim, both builder dry-runs, and the manifest facts the decisions rest on (the dtype tally, the null-position table, the 212-column comparison) were produced by that code or by ad-hoc pandas one-liners from the same author in the same environment. The second monoculture is a curl loop for the published-site probe. Confirm reader-facing outcomes without either: fetch URLs from a different tool, read files with a different reader, re-derive every count from the bytes, and run the validator only where a check says so — and then look for what it cannot see.

What landed

Repo Merged Issues Other writes
QuantEcon/data-lectures #124 (c4a286c, squash of two commits incl. Copilot fix a2feea8): rename + capture groups + nulls: blocks + dtype sweep + schema/AGENTS text, 24 files. #126 (818811b, squash incl. Copilot fix be76a6a): builders/_validate.py, scripts/validate_datasets.py, .github/workflows/validate-datasets.yml, both dynamic builders and _template.py rewired, countries.csv.yml and us_business_cycle_monthly.csv.yml contract fixes, 13 files. (#125, a one-line AGENTS.md change, merged between them and is not this session's.) Closed: #120 #121 #122 #113 (by #124), #119 (by #126), #14 (by hand, with a closing table). Decision comments on #120–#122, #113; a correction on #121; two revision comments on #118; stamp of #14 re-dated then closed No data bytes changed; no tags pushed anywhere
QuantEcon/status-projects #64: projects.yml data-lectures-automation → done, ended: 2026-09-07 Opened #67 (a "Recently closed" strip) —
QuantEcon/workspace-lectures — Comment on #40: first verification pass, 0 of 60 Track X rows cleared, per-host publish table, the .ml keep_files finding —
QuantEcon/lecture-wasm — Comment on #70: the four final URLs and a call-by-call mapping —

1. Reader-facing outcomes

The one thing a reader can see is a URL that stopped resolving. business_cycle_data.csv had been served publicly since 2025-02 (QuantEcon/data, then this repo) and the session renamed it on the strength of the manifest's consumers: [] — no org-wide sweep was run, which the workspace rules require before deleting anything another repo might read.

  • https://raw.githubusercontent.com/QuantEcon/data-lectures/main/lectures/business_cycle_data.csv answers 404 and …/lectures/gdp_growth_annual.csv answers 200 with sha256 41df23233ea238e4166c5c21cc383791015d4c9af6868b840901a8ce083c9da8 (the manifest's recorded value; compute it from the fetched bytes, not from the file in a clone).
  • Nobody reads the old name. gh search code 'business_cycle_data org:QuantEcon' (the token is distinctive, so search works here) returns only this repo's dated history notes (PLAN.md, migration.yml, scripts/audit_annotations.yml, the manifest header). Then the sweep search cannot do: a Trees-API pass over all ~277 org repos' default branches for the basename, including the translation repos, lecture-wasm's non-default branches (wasm, any repoint/*), course forks (tom-econ370-2025, 2026-tom-course) and .notebooks mirrors. Expected: zero readers outside this repo. Record the repo count swept. Swept 2026-09-07 (Sweep every QuantEcon repository for a reader of business_cycle_data.csv #134). Enumeration reconciles with the org: 215 public + 71 private = 286, so private and archived repositories are in scope and were swept. 283 of the 286 were swept twice — a recursive Trees-API pass over every default branch (34,179 entries, truncated: false on all 283, so no depth-1 fallback was needed) and a depth-1 clone with git grep for content, which is what covers the 26 archived repos and 3 forks that code search does not index. The 3 not swept (test-cli, numfocus, quantecon-book-dp) are empty — size: 0, zero branches, Trees returns HTTP 409 Git Repository is empty. Zero path hits for business_cycle_data anywhere in the org. The 14 content hits are all in this repository and all the dated history this box exempts (PLAN.md 27, 34, 301, 308, 309, 324, 343; AGENTS.md:58; builders/business_cycle.py:20; manifest-schema.yml 49 and 228 — a survival the checklist did not name; migration.yml:839; scripts/audit_annotations.yml:48; the gdp_growth_annual.csv.yml header). gh search code independently returned 9 lines, all from this repository. An adversarial re-run then closed three gaps beyond the box: every branch of the 8 audit repos and this one (268 branches, 43,118 entries) plus a full-history all-refs content grep of those 8 (883 refs, including lecture-wasm's 16 and its gh-pages) — the only path hits are 12, every one inside this repository on stale pre-rename branches; gh-pages org-wide (39 repos, 18,079 files), which no pass had covered and which is where the published site actually lives; and the org's only two submodules (both in lecture-mapping), which neither the Trees API nor a non-recursive clone expands. All zero. Residual, named rather than closed: non-default branches of ~245 low-risk repositories, fork PR heads outside the audit repos, and any out-of-org or non-GitHub reader.
  • The old URL at the previous org and repo names — github.com/QuantEcon/data/raw/main/lectures/business_cycle_data.csv and the pre-flatten path …/data/raw/main/business_cycle/business_cycle_data.csv if it existed — either 404 or redirect to a 404; neither serves bytes that would let a stale reader keep working silently.
  • The live audit dashboard https://quantecon.github.io/data-lectures/ still reports 40 static files, 40 migrated, 1 orphan, 23 live-API lectures (re-read from audit.json's stats, not the page), and CATALOG.md on main has a gdp_growth_annual.csv row and no business_cycle_data.csv row.
  • The status-projects home page https://quantecon.github.io/status-projects/ shows, under the infrastructure programme, a collapsed row reading "3 completed projects" and, expanded, data-lectures automation as "done September 2026"; the Pipeline page's done tab lists it newest-first above data-lectures scaffolding; its project page shows "ended 2026-09-07" and "4 of 4 work items closed" with no compliance findings (the scaffolding row carries three; the automation row must carry none). Reworded and confirmed 2026-09-07 (Decide how each of the eight open #127 boxes closes #133) against the live rendered DOM — these pages render from JSON client-side, so the served HTML carries none of this text. The collapsed row reads "▸ 3 completed projects": upstream-delta-register was set done/ended 2026-09-07 by QuantEcon/status-projects#53 at 00:41:34Z, 3h13m before PLAN: record the P3 data half, and retract a box that is now wrong #64 did the same here. Every other clause holds. Two wording notes: the visible work-items label is "all 4 closed" ("4 of 4 work items … closed" is its tooltip), and the done-tab tie order among the three 2026-09-07 projects rests on Array.prototype.sort stability over registry row order, not a documented rule.

2. Artifact integrity

  • Every data file under lectures/ on main hashes to its manifest's integrity.sha256: run python .github/scripts/check_consumed_files.py and an independent shasum -a 256 over the 44 files compared against grep sha256 lectures/*.yml. Expected 47 files hashed (44 datasets plus 3 under sources/ as LFS pointers), 0 mismatches. The two PRs claim no bytes changed: git diff --stat e318f06..818811b -- 'lectures/*' ':!lectures/*.yml' must show only the rename (R100).
  • The renamed pair is a pure rename: git log --follow --oneline lectures/gdp_growth_annual.csv reaches back to the 2026-07-16 flatten (Flatten the consumer-keyed tree into the published layout #10) and the blob hash at c4a286c equals the blob hash at e318f06 for the old path. Confirmed as written 2026-09-07 (Decide how each of the eight open #127 boxes closes #133) on a full clone: the box's own command returns 5 commits and reaches 52dbb89, the merge commit of Flatten the consumer-keyed tree into the published layout #10 (mergedAt 2026-07-16T22:45:04Z), then one further to b857c5c (2025-02-16); e318f06:lectures/business_cycle_data.csv and c4a286c:lectures/gdp_growth_annual.csv are the same blob 29250cae27e1422a69d0e42aabc7502ba08a14e5; the rename diff is exactly one R100 line. The verification comment's contrary finding — single root 931d626, 50 commits, no flatten commit — is a shallow-clone artefact: 931d626 is an ordinary commit with parent c044b8c and is precisely the 50th ancestor of 818811b, the head VALIDATION: independent review of the 2026-09-07 schema-decisions, rename and manifest-driven validator work #127 was written against. git clone --depth=50 there reproduces that finding bit for bit. The sole root is 77ece40 (2025-02-09); main carried 104 commits at 47017ea.

3. The highest-value claim to re-test: the validator catches what it says it catches

Every future refresh PR and every future manifest edit is gated by builders/_validate.py. The session proved it with a mutation script it never committed, on frames it built itself. Reproduce from scratch, in a fresh clone, without that script:

  • python scripts/validate_datasets.py on main under pandas 2.3.3 prints 44 manifest(s): 44 pass, 0 fail with conformance-only: {'xlsx': 5, 'npy': 2, 'dta': 4, 'json': 1, 'xls': 1}; the same under pandas 3.0.x (the lectures' anaconda=2026.07 pin). Record both pandas versions used.
  • Break one thing per rule, each as a one-line edit to a copy of the committed CSV or its manifest, and confirm the named failure: (a) append a column extra to gdp_growth_annual.csv → "unexpected column(s) not claimed"; (b) blank one Country cell in the same file → "Country: 1 nulls but not declared under known_nulls" (this is the Copilot-review fix in Manifest-driven validate(): the schema block is the spec, shared by the builders and a PR check #126, be76a6a); (c) blank DEU's YR2000 → "hole at YR2000 inside the series"; (d) blank DEU's YR2025 → passes (recent: 2); (e) blank UNRATE at 2020-05-01 in us_business_cycle_monthly.csv → "hole at 2020-05-01 inside the series is not declared under nulls.inner" — this one matters most, because Manifest-driven validate(): the schema block is the spec, shared by the builders and a PR check #126 removed the exact count that used to catch it, so placement alone must; (f) blank UNRATE at 2026-07-01 (the newest row) → passes (recent: 1); (g) blank INDPRO at 1919-01-01 → passes the shared validator (leading nulls allowed) and fails builders/business_cycle_fred.py's own validate() with "INDPRO: first observation is 1919-02-01"; (h) drop the YR1990 column → passes the shared validator (a regex cannot see a grid) and fails business_cycle.py's validate() with "gap in the year grid"; (i) set lingcod_msy_recovery.csv.yml's known_nulls to {F_over_Fmsy: 2} → "1 nulls, manifest says exactly 2". Cases (g) and (h) are the documented builder-layer margin; if either passes both layers, that is a regression.
  • The CI job actually gates: on a throwaway branch, commit mutation (a) and open a draft PR; validate-datasets must go red with a ::error file=lectures/gdp_growth_annual.csv.yml:: annotation in the checks tab, and consumed-file-check must go red too — mutation (a) rewrites the data file, and check_consumed_files.py hashes every file with a recorded sha256 whether or not consumers is empty. To see validate red while consumed-file-check stays green, use a manifest-only mutation such as (i). Close the PR without merging. Reworded and confirmed 2026-09-07 (Decide how each of the eight open #127 boxes closes #133) from the API: at 5e23ade (mutation (i)) validate failed (34085718978) and consumed-files succeeded (34085718933); at 93d0173 (mutation (a)) both failed (34085829077, 34085829084, the latter with gdp_growth_annual.csv: bytes do not match the manifest). The annotation path is confirmed through the check-runs API — gh run view --log renders ::error file=X::msg as ##[error]msg and drops the path, so the log alone cannot show it. Throwaway, do not merge: #127 §3.3 CI-gating check with a deliberate manifest mutation #130 is closed, not merged; its branch carries neither mutation.
  • The two builders still run end to end against the live sources, from a runner-like environment if possible (FRED stalls custom User-Agents from GitHub-hosted runners, builders/_fred.py docstring): python builders/business_cycle.py --out-dir /tmp/wb --summary-json /tmp/wb.json exits 0 and reports 0 of 585 cells revised for gdp_growth_annual.csv unless the World Bank has moved; python builders/business_cycle_fred.py --out-dir /tmp/fred --summary-json /tmp/fred.json exits 0 and its summary's overlap.new_columns lists the month(s) after 2026-07-01. Then python scripts/snapshots.py pr-body us_business_cycle_monthly.csv --summary /tmp/fred.json renders a title of the form Refresh us_business_cycle_monthly.csv: 2026-07-01 → 2026-0M-01.
  • The Monday canary is the real test the session could not run: the refresh-snapshots run of 2026-09-07 ~06:17 UTC (or the next one) has two green canary legs (one per builder) with the shared validator in the path, and the FRED leg's log shows the placement rule accepting the newest month's three blanks rather than failing on them. If it failed on "UMCSENT: N nulls" or "hole at 2026-0M-01", the recent: 1 fix did not reach the runner path.

4. Records written

  • All 44 sidecars, manifest-schema.yml and migration.yml parse with PyYAML; every filename equals its sidecar's name; every class: dynamic-snapshot manifest (4) has a nulls: block whose keys are a subset of {along, leading, recent, ended, inner}; no manifest has known_nulls_total at schema level (it survives only inside sheets[].known_nulls_total on assignat.xlsx.yml (4) and dette.xlsx.yml (5), each with read_as.header: null).
  • The dtype sweep was complete and did not over-reach: grep -h "dtype: " lectures/*.yml | sort | uniq -c shows no string, object or bare datetime; float32 remains only in the four .dta manifests; the 2 datetime64 in the hansen files were already there. Re-derive the sweep count the session claimed (21 string, 9 object, 2 datetime → 32 edits) from git diff e318f06 c4a286c -- lectures/.
  • The three capture-group patterns are 'YR(\d{4})' in gdp_growth_annual, unemployment_rate_annual, private_credit_to_gdp, and re.compile(p).groups == 1 for each; chapter_3.xlsx.yml's sheets_pattern: 'Table3\.\d+' was deliberately left alone (sheets, not columns) — confirm it still has no capture group and the conformance pass does not complain.
  • countries.csv.yml now declares Country code: 1 for Namibia: confirm with a reader that is not pandas (e.g. awk -F';' '$4=="\"NA\""' lectures/countries.csv or csvkit) that the bytes hold the string NA for Namibia and that pandas with keep_default_na=False yields 0 nulls in that column. Then check the two consumers named in the manifest — lecture-python-programming/lectures/pandas_panel.md and lecture-python.myst — for whether either uses Country code such that Namibia dropping out changes a figure or a table row on the published site.
  • forbes-billionaires.csv.yml now says government is bool: confirm the bytes hold True/False/empty (2606 empties of 2935 rows) and that its consumer lecture-python-intro (heavy_tails) never reads that column.
  • us_business_cycle_monthly.csv.yml says recent: 1 and known_nulls: {} and its comment gives first-vintage totals UNRATE 349, UMCSENT 616, CPILFESL 457, M0892AUSM156SNBR 1132; re-derive all four from the committed bytes with a non-pandas count of empty cells per column.
  • PLAN.md Phase 5's PR-validation box is ticked with a 2026-09-07 note and the two boxes below it (consumer fan-out, reusable workflow) are still unticked; manifest-schema.yml's header no longer says "nothing validates against it".

5. Tracker consistency

6. Known blind spots

  • Non-CSV formats are conformance-only (13 manifests: 5 xlsx, 4 dta, 2 npy, 1 json, 1 xls). Confirm the workflow log says so in words, and pick one — fp.dta (float32 columns) or dette.xlsx (positional ranges) — and check by hand that its manifest's dtypes/shape/known_nulls_total claims are true of the bytes, since nothing now or before does.
  • The published-site probe shares the settle policy's blind spot. The ws#40 comment says 0 of 60 rows cleared and attributes it to no publish tag since 2026-09-01. Re-derive: for each of the 8 hosts, the newest publish*-triggered run via gh run list --workflow publish.yml (tags are not date-ordered), and re-probe 5 rows per host with a different client (wget --spider or a browser). If any host published since 2026-09-01 and still serves, the cache theory needs revisiting. Ran 2026-09-07 (Re-probe the eight published hosts and record each host's newest publish run #135); the per-host table is on QuantEcon/workspace-lectures#40, which owns the rows. A census of all 60 rows, not a sample, on two clients that are not curl (wget, then Python urllib): 0 cleared, and every served body is byte-identical (sha256) to the pre-deletion git blob, so nothing cleared and nothing changed. Controls discriminate on all 8 hosts — eight distinct /intro.html bodies, one identical 9,379 B stock Pages 404. The 60 is now derived rather than asserted: the eight Track X commits removed 64 files, 60 of them under _static/lecture_specific/. The attribution is measured, not inferred — cache-buster GETs on a fresh CDN key returned identical bytes, and for the 3 gh-pages hosts the deleted file is still physically in the deployed tree. Seven hosts have not published since the deletions; .ml has deployed three times — including on the Track X commit itself — and still serves all six rows, which is positive proof of the structural keep_files: true exemption rather than an inference, so the cache theory is not impugned. Two things changed since the 00:30Z pass: the rebuild half of the gate is now met on all seven cache-bearing hosts (this morning's cache.yml runs), so only the tag is outstanding anywhere; and the deploy mechanism is now verified for all eight — quantecon/actions/publish-gh-pages is upload-pages-artifact + deploy-pages, a full replace, so every non-.ml host will prune on its next publish. Method note: gh run list createdAt is the run start, not the deploy; gh api repos/<r>/deployments?environment=github-pages is authoritative and works for all eight.
  • .ml never clears: lecture-python-programming.ml/.github/workflows/publish.yml deploys with peaceiris/actions-gh-pages@v4 and keep_files: true; confirm on the gh-pages branch that _static/lecture_specific/pandas/data/test_pwt.csv is still present in the tree, and check whether any other repo in the org deploys with keep_files: true (the session did not look).
  • The org sweep the rename skipped — item 1's second box — is the largest blind spot of the session; treat a hit there as a regression to file, not a caveat. Closed 2026-09-07 (Sweep every QuantEcon repository for a reader of business_cycle_data.csv #134): 286 repositories enumerated, 283 swept, zero hits outside this repository. No regression to file. See item 1's second box for the method.

7. Decisions settled today (re-checkable facts)

  • Under pandas 3.0.x a default read_csv yields dtype str for text and datetime64[us] with parse_dates; under 2.3.3, object and datetime64[ns]. Of 212 named CSV columns, 169 match exactly under pandas 3 and 142 under 2.3.3 before the sweep (at e318f06); after it, the family compare gives 44/44. Re-derive the 212 and the two match counts.
  • The FRED composite on 2026-09-07 ran to 2026-08-01 with only UNRATE and USREC populated in that month (the basis of recent: 1). Re-fetch https://fred.stlouisfed.org/graph/fredgraph.csv?id=UMCSENT,CPILFESL,INDPRO and check whether August has since been published; either way the rule stands. Re-measured 2026-09-07 (Record the date the FRED August publication lag closed #136) and confirmed: UMCSENT, CPILFESL and INDPRO all still end 2026-07-01, while UNRATE and USREC carry August — the exact state the rule was written against, and today's canary (run 34092578823) reports the same frame end independently. The lag had NOT closed when this issue was closed out, so there is no date to record; FRED's calendar puts the next releases at CPILFESL 2026-09-11, INDPRO 2026-09-18 and UMCSENT 2026-09-25, so the earliest full close is 2026-09-25. UMCSENT's 2026-08-28 release delivered no new observation at all (its ALFRED vintages for 08-27 and 08-31 are identical), so that date is the least reliable of the three. The date the lag closed is carried as an accepted residual, not a gate — the rule stands either way.
  • Namibia's ISO alpha-2 code is NA and pandas' default na_values includes the string NA (pandas docs, read_csv na_values).
  • The weekly cache.yml cron is 0 3 * * 1 UTC on lecture-python-programming, lecture-python.myst, lecture-python.zh-cn, lecture-python-programming.zh-cn; lecture-dp, .fr and .fa had push-triggered clean rebuilds on 2026-09-04.

8. Deliberately not done

Where the reasoning lives

AGENTS.md ("Naming a published file", "The schema block is executable", "Builders"), manifest-schema.yml (the comments beside filename, pattern, dtype, known_nulls, nulls), PLAN.md Phase 5 and Phase 8 P4, the decision comments on #120 #121 #122 #113 (with the #121 correction), the closing table on #14, the work plan #118, and the PR descriptions of #124 and #126 (each lists what it deliberately left out).

For the validator: work in a session that did not do this work. Do not use the tool named under "Bias to test for" except where a check explicitly says to run it. Re-derive counts rather than confirming them. Where a check can be run against a surface the original session did not exercise, do that too — the margin beyond the checklist is where regressions hide. Deliver: one comment on this issue with a per-item verdict (confirmed / confirmed with caveat / refuted / not completable, with evidence), a new issue for any regression found (do not bury findings in the comment), and leave the checkboxes to the issue owner unless told otherwise.

Activity

  1. quantecon-services
    added a commit that references this issue on Sep 7, 2026

    Throwaway for #127 section 3.3: mutation (a), an unclaimed column in …

    93d0173
  2. mmcky
    added a commit that references this issue on Sep 7, 2026

    Validation fixes: pandas-3 band check (#128), relative-path annotation (

    47017ea

  3. quantecon-services commented on Sep 7, 2026

    @quantecon-services
    quantecon-services
    on Sep 7, 2026 – with Claude
    Collaborator

    Independent verification, 2026-09-07 — a session that did none of the work

    Method. Fresh clone of main at 818811b. Every count re-derived from the bytes with sha256sum, awk, Python's csv module and PyYAML; the session's validator was run only where a box says to run it. Two throwaway environments built with uv for the pandas checks: pandas 2.3.3 and pandas 3.0.5 (both numpy 2.4.6, PyYAML 6.0.3, Python 3.11.15). URLs probed with three clients (curl, wget, Python urllib). Other repositories read from depth-1 git clones (32 public repos plus workspace-lectures and status-projects), never through gh search code.

    Limits of this environment, which account for every "not completable" below: the egress proxy denies quantecon.github.io, every *.quantecon.org host, api.github.com, fred.stlouisfed.org, api.worldbank.org and the Actions artifact store; gh is unavailable. The CI-gating draft PR (#130) was opened only after @mmcky authorised it, because the session's permission layer had refused to commit a deliberately broken data file unprompted. Verdicts: confirmed / confirmed with caveat / refuted / not completable. Two defects filed: #128, #129.

    1. Reader-facing outcomes

    Box Verdict Evidence
    Old URL 404, new URL 200 with the recorded sha256 confirmed …/lectures/business_cycle_data.csv → 404 from curl, wget and urllib. …/lectures/gdp_growth_annual.csv → 200, 10,628 bytes; sha256 of the fetched bytes 41df23233ea238e4166c5c21cc383791015d4c9af6868b840901a8ce083c9da8 (sha256sum and hashlib agree; equals the manifest).
    Nobody reads the old name confirmed with caveat In this repo git grep business_cycle_data hits only dated history: PLAN.md:27,34, migration.yml:839, scripts/audit_annotations.yml:48, AGENTS.md:58, builders/business_cycle.py:20, the manifest header. The Trees-API pass over ~277 repos could not run here. Substitute: depth-1 clones of 34 repos searched with git grep and git ls-files — the 8 audit repos, lecture-stats, five translation repos (lecture-python.zh-cn, lecture-intro.zh-cn, lecture-python-programming.{zh-cn,fr,fa,ml}), seven .notebooks mirrors, tom-econ370-2025, 2026-tom-course, test-actions-lecture-intro, python-lecture-sandpit.myst, QuantEcon.lectures.code, quantecon-notebooks-python, meta, website, skills, qeps, workspace-lectures, status-projects — plus all 15 branch tips of lecture-wasm (there is no wasm branch and no repoint/* branch; the non-default branches are datasets/track-a-remainder, track-x-orphans, ci/data-url-guard, gh-pages and eleven theme/docs branches) and .ml's gh-pages. Zero hits in content or paths outside this repository. lecture-dp.notebooks and lecture-python-intro.fa do not exist. The remaining ~240 org repos are unswept.
    Old org/repo URLs serve nothing usable confirmed raw.githubusercontent.com/QuantEcon/data/main/lectures/business_cycle_data.csv → 404; …/data/main/business_cycle/business_cycle_data.csv → 404. The raw host does follow the repo rename (…/QuantEcon/data/main/lectures/gdp_growth_annual.csv → 200), so those 404s are real. github.com/QuantEcon/data/raw/main/… (both paths) → 403 with a 378-byte body and no redirect; through the same proxy github.com/QuantEcon/data-lectures/raw/main/lectures/gdp_growth_annual.csv → 302 → 200, so the 403 is GitHub's answer. Nothing serves bytes under the old names.
    Dashboard 40 / 40 / 1 / 23 and the CATALOG rows confirmed with caveat Pages host denied here, and the github-pages artifact of run 34080816745 (13.3 MB) sits on a blob host the proxy also denies. Re-derived instead: build_audit.py scan --strict over fresh depth-1 clones of the 8 lecture repos → static_files 40, migrated 40, orphans 1, api_lectures 23, generated 2026-09-07; the run that built the live site (818811b, 03:47 UTC) succeeded through its deploy job. CATALOG.md on main has one gdp_growth_annual.csv row (line 35) and no business_cycle_data.csv row.
    status-projects home / pipeline / project page refuted as stated Registry and the 2026-09-07T04:19Z collection: data-lectures-automation is done, ended 2026-09-07, tracker closed, 4 of 4, findings: []; data-lectures-scaffolding carries no-sub-issues, unstamped, untyped. But the infrastructure programme has three done projects, not two: upstream-delta-register was set done / ended 2026-09-07 in status-projects #53 (00:41 UTC, "one repoint, one priority, two done"), three hours before #64. The collapsed row therefore reads "3 completed projects". The done tab sorts by declared.ended descending, so automation sits above scaffolding, tied on the date with upstream-delta-register and lecture-proposals. Live pages not fetched (host denied).

    2. Artifact integrity

    Box Verdict Evidence
    Hashes confirmed sha256sum over the 44 data files against the manifests' sha256: lines: 0 mismatches. check_consumed_files.py: 44 manifests, 47 files hash-checked (44 + 3 LFS pointers under sources/), 0 errors. git diff --name-status -M e318f06..818811b -- 'lectures/*' ':!lectures/*.yml' → exactly one line, R100.
    Pure rename back to #10 confirmed with caveat Blob 29250cae27e1422a69d0e42aabc7502ba08a14e5 at e318f06:lectures/business_cycle_data.csv, at c4a286c and at 818811b:lectures/gdp_growth_annual.csv. But git log --follow reaches 931d626 (2026-08-10, #57) and stops: this repository's history has a single root, 931d626, 50 commits in total. No commit for the 2026-07-16 flatten (#10) exists in it, so the file's earlier history is recorded only in prose.

    3. The validator catches what it says it catches

    Box Verdict Evidence
    44 / 44 under both pandas confirmed pandas 2.3.3 and 3.0.5 both print 44 manifest(s): 44 pass, 0 fail; bytes-validated formats: csv; conformance-only: {'xlsx': 5, 'npy': 2, 'dta': 4, 'json': 1, 'xls': 1}. CI job 101615598693 on 818811b (pandas 2.3.3, Python 3.12) printed the identical line.
    Mutations (a)–(i) confirmed, one caveat Edited with awk or the csv module in a fresh clone and run through the committed CLI under both pandas versions; every named failure appears verbatim. (a) unexpected column(s) not claimed by any entry: ['extra']. (b) Country: 1 nulls but not declared under known_nulls. (c) DEU: hole at YR2000 inside the series is not declared under nulls.inner. (d) passes. (e) UNRATE: hole at 2020-05-01 inside the series is not declared under nulls.inner. (f) passes. (g) shared validator passes; business_cycle_fred.validate() fails INDPRO: first observation is 1919-02-01 00:00:00, not 1919-01-01. (h) shared validator passes; business_cycle.validate() fails gdp_growth_annual.csv: gap in the year grid. (i) F_over_Fmsy: 1 nulls, manifest says exactly 2. Neither (g) nor (h) passes both layers. Margin: blank DEU YR2023 → fails (the recent: 2 boundary is exact); blank UMCSENT at 2026-06 → fails (recent: 1 boundary exact); blank DATE cell → DATE: 1 nulls but not declared under known_nulls; blank USREC in the newest month → shared validator accepts (recent covers every series), builder catches USREC has a missing month; a dropped economy → 8 rows, floor is 9; YR2000→YR200X → unclaimed columns; the 2026-09-07 live shape (2026-08 appended with UMCSENT, CPILFESL, INDPRO blank; previous = committed) → passes with new_columns ['2026-08-01']; two lagging months → fails as designed. Caveat: with a relative manifest path — the form the script's own usage line shows — a failing run dies in ValueError at path.relative_to(REPO) instead of printing the ::error line (#129).
    The CI job gates a PR confirmed, with one correction Draft PR #130, opened on a throwaway branch and closed unmerged; the branch was then reset to main. Head 5e23ade, mutation (i) alone (manifest says F_over_Fmsy: 2): validate failure (run 34085718978) — the log carries ##[error]F_over_Fmsy: 1 nulls, manifest says exactly 2, FAIL lingcod_msy_recovery.csv.yml: 1 problem(s), 44 manifest(s): 43 pass, 1 fail; consumed-files success (run 34085718933); audit-dashboard build success. Head 93d0173, mutation (a) added (an extra column in gdp_growth_annual.csv): validate failure (run 34085829077) — ##[error]unexpected column(s) not claimed by any entry: ['extra'], FAIL gdp_growth_annual.csv.yml: 1 problem(s), 42 pass, 2 fail; and consumed-files failure (run 34085829084) — gdp_growth_annual.csv: bytes do not match the manifest (sha256 384d6d98… != recorded 41df2323…). So the schema gate does block a PR, but the box's "consumed-file-check stays green" holds only for a manifest-only mutation: check_consumed_files.py hashes every file with a recorded sha256 regardless of consumers, so mutation (a) turns both checks red. (The runner renders the CLI's ::error file=lectures/….yml:: command as the ##[error] line above and attaches the file to the checks-tab annotation.)
    Builders end to end against live sources not completable here World Bank and FRED hosts denied. Offline substitute: both builders' own validate() on the committed bytes as both current and previous → FRED overlap 0 cells revised, new_columns []; under pandas 2.3.3 all four tables pass. Under pandas 3.0.5 the World Bank builder rejects its own committed snapshots with value out of band on all three tables — #128 (pre-existing since #114, gated only by the requirements.txt pin). snapshots.py pr-body not exercised (needs a live summary).
    Monday canary pending The cron is 17 6 * * 1. As of posting (05:25 UTC) the workflow has exactly four runs, all workflow_dispatch, the newest 2026-09-01 06:36 UTC (success on a0e1415); no scheduled run has fired yet. The 06:17 UTC run's two legs, and whether the FRED leg's log shows the placement rule accepting the newest month's blanks, will follow in a short comment once it reports.

    4. Records written

    Box Verdict Evidence
    Parse, filenames, nulls: keys, known_nulls_total confirmed PyYAML parses all 44 sidecars, manifest-schema.yml and migration.yml; every filename equals its sidecar; the 4 dynamic manifests each carry exactly {along, ended, inner, leading, recent}; no schema-level known_nulls_total; it appears 4× in assignat.xlsx.yml and 5× in dette.xlsx.yml, every entry under read_as.header: null.
    dtype sweep confirmed Tally on main: float64 126, str 72, float32 57, int64 49, datetime64 4, bool 1 — no string, object or datetime; float32 only in fp, maketable1/2/4.dta.yml; the 2 hansen datetime64 lines are already there at e318f06. git diff e318f06 c4a286c -- lectures/ dtype lines: −21 string, −9 object, −2 datetime (the 32 edits), plus str→bool for forbes government and 3 float64 pattern lines rewritten for the capture group; +30 str, +2 datetime64, +1 bool, +3 float64.
    Capture groups confirmed 'YR(\d{4})' in the three manifests, re.compile(p).groups == 1 for each; chapter_3.xlsx.yml sheets_pattern: 'Table3\.\d+' has groups == 0 and reports "conformance only (xlsx)".
    Namibia confirmed awk -F';' on the bytes: row 149, column 4 is the quoted string NA (Namibia), the only such cell; STR_NA_VALUES contains 'NA' in 2.3.3 and 3.0.5; default read → 1 null (Namibia); keep_default_na=False → 0 nulls, code 'NA'. pandas_panel.md in lecture-python-programming, lecture-python.myst and lecture-python.zh-cn (and the .zh-cn/.fr/.fa programming translations) never mentions Country code, so no figure or table row changes.
    forbes government confirmed csv module: 2,935 rows; government = 2,606 empty, 328 False, 1 True. heavy_tails.md in lecture-python-intro (also lecture-wasm, lecture-intro.zh-cn, test-actions-lecture-intro) never reads the column.
    FRED composite confirmed Manifest: recent: 1, known_nulls: {}, the four totals in the comment. Empty cells by the csv module (UNRATE cross-checked with awk): UNRATE 349, UMCSENT 616, CPILFESL 457, M0892AUSM156SNBR 1132; 1,291 rows, 1919-01-01 .. 2026-07-01.
    PLAN / schema header confirmed Phase 5 PR-validation box [x] with "Landed 2026-09-07 (#119)"; fan-out and reusable-workflow boxes [ ]; manifest-schema.yml says "since 2026-09-07 the schema block IS validated" and the phrase "nothing validates against it" is gone.

    5. Tracker consistency

    Box Verdict Evidence
    #14 confirmed Closed completed 2026-09-07T03:49:42Z; sub-issues #119 #120 #121 #122 all closed (4 of 4); body heading "Where we stand (verified 2026-09-07)"; #60 #109 #110 #112 #114 #117 #124 #126 all exist with merged_at set.
    CATALOG current confirmed build_catalog.py then git diff --exit-code CATALOG.md → exit 0.
    Workflows green, generated confirmed with caveat On 818811b: validate-datasets 34080816780, consumed-file-check 34080816788, audit-dashboard 34080816745 — all success; the Pages artifact was produced 03:47:29Z. Live audit.json unreachable; the local rebuild's generated is 2026-09-07.
    status-projects latest.json confirmed with caveat Collection generated_at 2026-09-07T04:19:44Z: automation declared.stage done, declared.ended 2026-09-07, observed.tracker.state closed, progress {completed: 4, total: 4}, compliance.findings []. datasets-migration is still active — but with findings: [] and tree.coverage: spans-start; no snapshot from 09-01 to 09-07 carries tree-postdates-start (the tree fields first appear in the 09-07 collection).

    6. Known blind spots

    Box Verdict Evidence
    Non-CSV conformance-only; one checked by hand confirmed The CI log prints conformance only (xlsx) (etc.) per manifest and bytes-validated formats: csv; conformance-only: {…} in the summary. fp.dta via read_stata: 5,432 × 36, columns in the declared order, all 33 known_nulls counts exact, dtypes match (country declared str, read object, same family). dette.xlsx via read_excel with each sheet's read_as: all five ranges match shape and known_nulls_total (102×12 / 0, 105×1 / 32, 99×2 / 78, 15×1 / 0, 120×1 / 8), all float64 as declared.
    Published-site probe not completable here (partial) All 8 hosts denied by the proxy; gh run list unavailable. Independent partial: newest publish* tag by creatordate per repo — programming publish-2026aug20, python.myst publish-2026aug14, dp publish-2026jul07 (dp also publishes by workflow_dispatch, which leaves no tag), python.zh-cn publish-2026aug24b, programming.zh-cn / .fr / .fa publish-2026aug19; .ml publishes on push, last gh-pages deploy 2026-09-04. No publish tag after 2026-09-01 on any host, consistent with the ws#40 pass.
    .ml keep_files confirmed publish.yml:85 keep_files: true, with the preview-directory rationale at line 83; gh-pages at cf37341 (2026-09-04 deploy) still holds _static/lecture_specific/pandas/data/test_pwt.csv, beside 5 pr-* directories. None of the other 31 public repos swept mentions keep_files in its workflows.
    Org sweep see §1 Zero hits in 34 repos and 15 lecture-wasm branch tips; ~240 repos unswept.

    7. Decisions settled today

    Box Verdict Evidence
    dtype names; 212 / 169 / 142 confirmed with caveat pandas 3.0.5 default text dtype str, datetime64[us] with parse_dates; 2.3.3 object, datetime64[ns]. Named CSV columns at e318f06: 212. Exact str(dtype) matches (delimiter honoured, parse_dates on declared datetime columns, name: "" mapped to Unnamed: k): 150 under 2.3.3 and 177 under 3.0.5 — eight more than claimed in both versions, so the offset lies in the numeric columns (which match identically under both) and is a counting convention, most likely index-column handling, not a disagreement about the text columns the decision rests on. After the sweep, at main: 141 vs 206 exact; the family compare gives 44 / 44 under both.
    FRED lag closed? not completable Host denied.
    Namibia / NA confirmed The bytes hold NA; pandas._libs.parsers.STR_NA_VALUES includes 'NA' in both versions.
    Crons confirmed with caveat cache.yml cron 0 3 * * 1 in lecture-python-programming, lecture-python.myst, lecture-python.zh-cn, lecture-python-programming.zh-cn (also intro, advanced, jax, continuous_time_mcs, stats); lecture-dp 0 2 * * 1. .fr and .fa have a cache.yml triggered on push to main with no cron, and both HEADs are 2026-09-04 — the mechanism for a 09-04 push-triggered rebuild is there; the runs themselves are unverifiable without the API.

    8. Deliberately not done

    Box Verdict Evidence
    #87 renames not made confirmed fred_data.csv, test_pwt.csv, acs_data_summary.csv, dataBHS.csv all present on main.
    PLAN boxes confirmed Both unticked; #14's closing comment records where they belong.
    Mutation script / required check confirmed with caveat No test or mutation script in git ls-files; #126's "Not in this PR" records both. Branch protection is not readable with this session's tools.
    History notes confirmed PLAN.md:27,34, migration.yml:839, scripts/audit_annotations.yml:48, plus AGENTS.md:58, builders/business_cycle.py:20 and the manifest header.

    Checkbox status

    At @mmcky's request the 25 boxes whose claim held as written are ticked in the body above (some with the caveats noted in their rows). Ten stay open, in three kinds: the box's expectation is wrong even though the facts are fine — §1.5 (three completed projects, not two), §2.2 (--follow cannot reach #10; the blob-hash half holds), §3.3 (the gate works, but consumed-file-check does not stay green for mutation (a)), §5.4 (tree-postdates-start is absent; the automation half holds); the check needs a network this session does not have — §1.2 and §6.4 (the org-wide Trees-API sweep; 34 repos and 15 lecture-wasm branch tips are clean, ~240 repos unswept), §3.4 (live builders), §6.2 (site re-probe), §7.2 (FRED lag); and §3.5, pending the 06:17 UTC canary.

    Regressions and margin findings

    Generated by Claude Code

  4. quantecon-services commented on Sep 7, 2026

    @quantecon-services
    quantecon-services
    on Sep 7, 2026 – with Claude
    Collaborator

    Canary result (§3.5), and the live-builder box (§3.4)

    The scheduled refresh-snapshots run fired at 06:49:58 UTC (cron 17 6 * * 1; GitHub's scheduler was 33 minutes late): run 34092578823 on main at 47017ea — after #131 merged the fixes for #128 and #129 at 05:30 UTC, so the canary ran the corrected builders. Conclusion success; the notify and refresh jobs were skipped, as they should be when both legs pass and nothing is due.

    Leg Job Result
    canary (gdp_growth_annual.csv) — builders/business_cycle.py 101649080627 success. Log: gdp_growth_annual.csv: overlap YR1960..YR2025: 0 of 585 cells revised, max |change| 0.000; new columns none; new economies none; unemployment_rate_annual.csv 0 of 243 and private_credit_to_gdp.csv 0 of 65, likewise; all three written 1960..2025 to canary-out.
    canary (us_business_cycle_monthly.csv) — builders/business_cycle_fred.py 101649080629 success. Log: overlap window 1919-01-01..2026-07-01: 0 cells revised; max |change| by series all 0.0; wrote us_business_cycle_monthly.csv: 1292 months x 6 series (1919-01-01 .. 2026-08-01). The live frame carries one new month, 2026-08-01, so overlap.new_columns is ['2026-08-01'].

    The placement rule. The builder's log does not print per-series null counts, so the acceptance shows as the pass itself: a blank the rule rejects prints <series>: null at 2026-08-01 after its last observation … and exits 2, which would have failed the leg and fired notify; neither happened. Given the release calendar (CPILFESL and INDPRO around the 15th, UMCSENT's final at month end), UMCSENT, CPILFESL and INDPRO were unpublished for August at 06:50 UTC on the 7th, so the three blanks were present and accepted under recent: 1. builders/_validate.py is untouched by #131, so this is the rule #126 shipped.

    pr-body. scripts/snapshots.py is unchanged by #131. Fed a summary of exactly this live shape (previous = the committed bytes, new = committed + 2026-08 with only UNRATE and USREC populated, produced by the same validate() that writes the canary's summary.json), it renders Refresh us_business_cycle_monthly.csv: 2026-07-01 → 2026-08-01, with New columns | 2026-08-01 in its table.

    Verdicts. §3.5 confirmed: two green legs, the shared validator in the path, the newest month's blanks accepted. §3.4 confirmed with caveat: both builders ran end to end against the live sources from a GitHub-hosted runner with the expected 0 of 585 and the month after 2026-07-01, but the run was the canary's on 47017ea rather than a hand-run, and its summary.json is not retained on success, so the pr-body step was exercised on an equivalent summary. Both boxes ticked; eight stay open (§1.2 and §6.4, §1.5, §2.2, §3.3, §5.4, §6.2, §7.2), for the reasons in the comment above. #128 and #129 are closed by #131, which also puts a pandas 2.3.3 / 3.0.5 matrix and a --builders pass into validate-datasets, closing the gap that let #128 through.

    Generated by Claude Code

  5. mmcky
    added a commit that references this issue on Sep 7, 2026

    Record two repository facts the 2026-09-07 validation got wrong (#141)

    02c220d

  6. mmcky commented on Sep 7, 2026

    @mmcky
    mmcky
    on Sep 7, 2026
    ContributorAuthor

    Closing: every one of the 35 boxes now has a verdict, and all 35 are ticked. Twenty-seven were ticked on the two verification comments above; the remaining eight were settled today in #133, five of them by running the check rather than waiving it — this session had the network and gh the validating session lacked.

    How the eight open boxes closed

    Box Disposition
    §1.2 + §6.4 org-wide sweep ran — 286 repositories enumerated, 283 swept, zero hits outside this repository
    §6.2 published-site re-probe ran — all 60 rows, 0 cleared, mechanism measured rather than inferred
    §7.2 FRED lag confirmed and reworded — the lag had not closed; the date is an accepted residual
    §1.5 status-projects pages reworded — the live page reads "3 completed projects", not 2
    §3.3 CI gate reworded — consumed-file-check stays green only for a manifest-only mutation
    §5.4 latest.json reworded — findings: [] and tree.coverage: spans-start
    §2.2 pure rename ticked as written — the refutation was wrong; see below

    The validation itself had three errors, and one of them nearly propagated

    Re-measuring the four "wrong as written" boxes rather than accepting the verification comment's word changed three of the four verdicts.

    §2.2 was not wrong. The comment refuted it on the grounds that this repository "has a single root, 931d626 (#57, 2026-08-10), 50 commits in total", so no commit for the 2026-07-16 flatten exists. Every clause of that is false. 931d626 is an ordinary commit with parent c044b8c, with 54 commits before it; the sole root is 77ece40 ("Initial commit", 2025-02-09); main carried 104 commits at 47017ea; and the box's own command reaches 52dbb89, the merge commit of #10, mergedAt 2026-07-16T22:45:04Z. The cause is a shallow clone: 931d626 is precisely the 50th ancestor of 818811b, the head #127 was written against, and git clone --depth=50 there reproduces the finding bit for bit, because a shallow boundary hides its parents and so is indistinguishable from a root.

    This was on its way into the repository. #138 was scoped to write "the git history begins at #57" into AGENTS.md as a durable fact, and #139's checklist lessons repeated it. Both were corrected before merging; what landed instead is the true statement plus the check that would have caught it.

    §5.4's refutation was also wrong, in the opposite direction. The comment said no 2026-09 snapshot carried tree-postdates-start on datasets-migration. Replaying the 47 collections in status-projects data/history/: the flag was introduced by collector commit 57db2e3 at 03:40 UTC on 2026-09-07, datasets-migration carried it for exactly three collections, and it cleared at 04:19:44Z. The validating session read the 04:19Z collection, eighteen minutes later. Both the box and its refutation were wrong about the same field.

    §1.5's refutation was right and is now confirmed against the live rendered DOM rather than the registry alone.

    Where every finding landed

    Accepted residuals

    Named here so nothing rides on this issue staying open.

    • The date the FRED August lag closed. Not measurable on 2026-09-07 — UMCSENT, CPILFESL and INDPRO all still ended 2026-07-01. FRED's next releases are CPILFESL 2026-09-11, INDPRO 2026-09-18, UMCSENT 2026-09-25, so the earliest full close is 2026-09-25. The recent: 1 rule is confirmed either way; the date is a record, not a gate.
    • The 60 Track X rows still served, and the .ml gh-pages prune. Both belong to QuantEcon/workspace-lectures#40. .ml deploys with keep_files: true, so its six paths can never clear by publishing and need a one-off prune; the other seven clear on their next tag.
    • Three unswept repositories — test-cli, numfocus, quantecon-book-dp — all empty, zero branches, no files.
    • Sweep coverage not attempted: the non-default branches of roughly 245 low-risk repositories, fork PR heads outside the eight audit repositories, and any out-of-org or non-GitHub reader (a CDN, a published bundle, a local clone). The last is unmeasurable in principle, so the sweep is a bound there rather than a proof.
    • The settle theory has no positive demonstration yet. No non-.ml host has published post-deletion, so while the deploy mechanism is now read from source for all eight and says every non-.ml host will prune, the decisive experiment — publish one site, re-probe — has not run. lecture-dp is the cleanest candidate to force, two months stale with a fresh cache.
    • A ticked box carries an imprecision, left alone because Close out the 2026-09-07 validation (#127) #132 puts the 27 ticked boxes out of scope: §7.4 says lecture-dp had a "push-triggered clean rebuild on 2026-09-04", but dp's cache.yml is schedule: cron '0 2 * * 1' (not the 0 3 * * 1 the same box states) plus workflow_dispatch plus a push restricted to paths: [environment.yml]. Its 2026-09-07 rebuild came from the schedule.

    For the next validation checklist

    1. A depth-limited clone reports its shallow boundary as a root commit. git rev-list --max-parents=0 returns the boundary and git log --follow stops there. Run git rev-parse --is-shallow-repository before concluding history is missing, and cross-check a suspected root against the API — a true root has parents: []. A tell-tale: the commit count exactly equals a round clone depth.
    2. The hash gate covers every recorded file, not only consumed ones. Phrase any "one check red, the other green" expectation against a manifest-only mutation.
    3. An expectation about another repository's dashboard needs the collection timestamp it was read at. Both §5.4 and §1.5 turned on values that changed within hours; §5.4's flag existed for 21 minutes. State the snapshot, not just the value.
    4. Re-measure a refutation before acting on it. Three of the four boxes this issue left open as "wrong as written" had a different verdict on re-measurement, and one refutation was about to be written into AGENTS.md as a durable fact.

    Generated by Claude Code

  7. mmcky
    closed this as completedon Sep 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions