VALIDATION: independent review of the 2026-08-07 french_rev / write-back work #51

Activity

1. mmcky commented on Aug 7, 2026

   

   mmcky

   on Aug 7, 2026

   ContributorAuthor

   More actions

   Update before validation begins — section 6's regression is resolved, and it grew a second finding.

   The regression this issue points to (QuantEcon/workspace-lectures#25) is now closed. All three translation sync PRs are merged — QuantEcon/lecture-intro.zh-cn#285, #286, #288 — and zh-cn's main no longer references lecture-python-intro/…/lectures/datasets/ anywhere; all eight datasets serve 200 from data-lectures.

   Two adjustments to section 6, both of which make it a stronger check rather than a weaker one:

   The validator should still run the org-wide sweep. The cause is unchanged: the pre-deletion sweep covered manifest.yml's nine repos and was described in QuantEcon/lecture-python-intro#827 as "org-wide". The fact that the symptom is fixed says nothing about whether lecture-intro.zh-cn was the only repo affected — that was never established, and the follow-up check only looked at repos whose names match intro|wasm, which is itself a guess.

   A new blind spot to verify, found while fixing #285. The sync PR overwrote a deliberate localisation: it replaced this repo's code_to_name = pd.read_csv("../lectures/datasets/country_code_cn.csv") with the English source's version, dropping the name_chinese column that nine call sites depend on. Had it built green, every chart in long_run_growth would have reverted to English country labels on the Chinese site. It failed loudly only by luck — a downstream cell indexes name_chinese by name and raised KeyError; a localisation affecting only rendering would have merged silently.

   Worth adding as checks:

   • Confirm no check anywhere detects an overwritten localisation, in either repo
   • Confirm the collision is intermittent — #286 and #288 touched URL lines only and merged clean, so "the last sync was fine" is not evidence about the next
   • Confirm the restored line is live on zh-cn main (long_run_growth.md:141) and that :263 still reads name_chinese
   • Beyond what was tested: sweep the remaining Track A-D lectures with translated counterparts for localised lines adjacent to data reads, and report which future sets will collide

   This strengthens rather than weakens the "Bias to test for" framing at the top: the session's sweep was scoped to its own workspace, and both findings in this section came from looking outside it.
2. mmcky commented on Aug 7, 2026

   

   mmcky

   on Aug 7, 2026

   ContributorAuthor

   More actions

   Independent validation — per-item verdicts

   Run in a session that did none of the 2026-08-07 work, against data-lectures a935449, lecture-python-intro 7694e3ec and lecture-wasm eceed27 (all clean, all at origin/main). Read-only throughout: no writes under any repo, no GitHub mutations. Where a repo's own script had to run, it ran against a cp -a copy in scratch.

   Result: 26 confirmed, 18 confirmed-with-caveat, 3 refuted, 1 not-completable. 45 candidate findings were raised; each was put through an adversarial pass whose job was to refute it, and 21 did not survive — they are not reported here.

   On the named bias

   The instruction was to avoid the session's fetch-script-plus-curl pair. The substantive checks were re-derived by methods that pair cannot produce:

   Method	Used for
   Local git object store — cat-file, blob-SHA identity, entirely offline	byte-identity, prose repointing, orphan baselines
   Real headless Chromium executing Pyodide	the wasm lecture, all 22 cells
   SHA-256 computed inside the browser from the page's own origin	CORS and host routing
   GitHub Actions build artifacts, pre- vs post-migration	figure pixel-identity
   Local socket servers	the pandas storage_options claim
   GitHub Trees API over all 277 org repos	the "last copy in the org" claim
   Shallow clone + grep of all 253 non-archived org repos	the org-wide sweep

   Three checks could not be run as written and were substituted, declared here rather than quietly: Colab (needs an interactive Google sign-in — substituted execution from a bare directory with no datasets/, which is the condition Colab creates); the CI-invisibility test (would have required pushing a scratch branch — substituted a local injection into a scratch clone, with a positive control); Wayback (see §1).

   ---

   1. Reader-facing outcomes

   Check	Verdict
   french_rev.html renders, every figure present	confirmed
   Notebook runs end to end	confirmed
   Colab	confirmed-with-caveat — substituted, see above
   Three Data Sources links reach data-lectures	confirmed-with-caveat
   Wayback pixel comparison	not-completable
   Beyond what was tested: wasm in a browser	confirmed

   18 <img> in <main>, all complete, none broken — and the count is right: 15 plt.figure() + 3 plt.subplots() = 18 plt.show() = 18 images, so no figure silently vanished. Against a locally generated blank-axes control (nonwhite_frac 0.0183, 0 coloured pixels), every published figure sits far above it (0.033–0.078 nonwhite, 338–1853 distinct colours). Four figures were re-rendered from the data-lectures bytes and matched to the published PNGs by block-density correlation: 0.83–0.90, against 0.06–0.12 for unrelated-figure controls.

   The notebook executes clean from a bare directory with no datasets/ — which is the actual substance of the Colab check. 60 cells, all data cells succeed.

   The wasm check — the surface the session never exercised — passes, and passes properly. All 22 cells ran in a real Pyodide runtime (kernel ready ~5 min). Every dataset fetch returned 200 from raw.githubusercontent.com, zero CORS console errors, and SHA-256 computed inside the browser matches the git blobs for all five files. The negative control is what makes this meaningful: the other host form, github.com/QuantEcon/data-lectures/raw/…, throws TypeError: Failed to fetch with an explicit CORS block from that same origin. So rule 5's host distinction is real, and wasm is on the correct side of it.

   Wayback is not-completable by construction, not by accident. The Wayback Machine holds french_rev.html (9 captures) but has never archived a single one of its figure PNGs — CDX returns empty for all 15, and id_ fetches 404 on 15 of 15. The newest pre-migration capture is also 2026-03-09, five months stale and predating two commits that changed the figure set, so it would have been a contaminated baseline regardless. The substantive claim was re-derived another way and holds: the build artifact from publish-2026aug06 (last pre-migration build) and from the migration commit reference an identical list of 18 _images/*.png, and all 18 are sha256-identical. Jupyter Book names each PNG by the sha256 of its own bytes, so identical filenames are byte identity.

   The Data Sources caveat: the three links are correct, 200, no redirects, and resolve to blobs that match git cat-file — but only three of the five migrated datasets are documented. caron.npy and nom_balances.npy are fetched at french_rev.md:711-712 and appear nowhere in the prose. Nine of the eighteen figures derive entirely from those two files. The checklist's own wording ("the three Data Sources links") bakes the gap in.

   2. Artifact integrity

   Check	Verdict
   sha256 vs manifest, all five	confirmed
   Git-history triangulation at 6986e35a~1	confirmed
   Recorded sizes	confirmed
   read_excel frames DataFrame.equals	confirmed-with-caveat
   Read arguments vs manifest read_as	confirmed-with-caveat
   .npy shapes and dtype	confirmed

   This section is the strongest in the issue. Byte-identity survives four independent chains — offline git blob identity, the contents API, the raw CDN, and the github.com/*/raw/ form the lecture actually uses — and all four agree on all five files. 6986e35a was verified to be #826's merge commit, touching only lectures/french_rev.md, so ~1 genuinely is the pre-repoint tree. Sizes match across git cat-file -s, wc -c and the API. Both arrays are exact: (63,2) and (81,2), float64, bitwise-identical old blob vs live, and the load-bearing cross-file alignment nom_balances[14:77,1] * caron[:,1] reproduces.

   The caveat is a counting error, not a substance error: there are 13 read_excel calls, not ten — ten distinct argument sets, three of which are used twice. All 13 return .equals() == True, under both pandas 2.3.3 and 3.0.5. Manifest read_as entries map 1:1 onto the ten distinct sets with nothing undocumented and nothing unused.

   3. Repoint rule 6

   Check	Verdict
   media 404 / raw 200 on a plain-git file	confirmed
   The converse — LFS file, media serves, raw returns pointer	confirmed
   Both hosts send access-control-allow-origin: *	confirmed
   14 affected reads, and the PLAN.md table	confirmed-with-caveat
   Invisible to CI	confirmed-with-caveat

   Rule 6 now rests on n=12 rather than n=1. Seven plain-git files 404 on the media host — including two inside high_dim_data itself, which proves this is per-path LFS tracking and not a repo-level property — and five LFS files serve real bytes on media while raw returns 130-byte pointer text with a 200, a silent parse failure rather than an error. Both hosts send access-control-allow-origin: *, including on the 404, and a browser fetch from a real page origin reads every status without throwing: the media 404 is a genuine routed 404, not a CORS block. Host routing, as rule 6 says.

   The count of 14 reproduces exactly, and every file:line in the PLAN.md table matches byte-for-byte. The builder-notebook question checks out: build_audit.py:216 selects only lectures/**.md and :222 excludes /_static/, and audit.json lists exactly the 14 minus those two.

   Both caveats are scope, not correctness. 14 is workspace-scoped — lecture-intro.zh-cn carries 7 more live media-host reads that the step-3 fold will also have to move. And the CI claim is confirmed with one correction: build_audit.py scan --strict exits 0 with a media-host URL injected (while cheerfully recording url_form: media, lfs_media: True and never asserting on it), and check_consumed_files.py exits 0 because it never looks at consumer URLs at all — but the framing "no check anywhere" overstates slightly, since intro's ci.yml does execute lectures/*.md, so an actually-broken read would fail there even though the host defect itself is invisible. The positive control confirms the negative is real: flipping wasm to github.com/*/raw/ gives exit 1 with 5 rule-5 warnings.

   4. Records written

   Check	Verdict
   Five manifests parse, filename matches sidecar	confirmed-with-caveat
   Exactly two consumers each, and they really read them	confirmed
   positional_reads on the three workbooks, not the .npy	confirmed
   18/18/0 errors, risen from 13, hashes bytes	confirmed-with-caveat

   All 18 sidecars parse, all filenames match, no duplicate keys. Consumers verified in both directions, including by execution against both URL forms. The 13→18 delta was re-derived structurally from git show at four revisions and is exactly the five french_rev files. Hashing is real — flipping one byte of caron.npy produces bytes do not match the manifest and exit 1.

   Two caveats worth carrying forward. The parenthetical rationale is wrong: a filename typo is not silently invisible — it produces ::error:: … does not match the sidecar's own name and exit 1 (verified by mutation), and only build_audit.py keys on filename; build_catalog.py returns a list. More materially, check_consumed_files.py hashes the repo's own committed copy against the manifest — no URL, no host, no network. So "0 errors" says nothing about what the lectures actually fetch; the exact class of regression rules 5 and 6 exist to prevent would leave it green. And "18 manifests" is 18 of the 19 datasets in lectures/ — business_cycle_data.csv has no sidecar and is invisible to the check.

   positional_reads is absent on the two .npy manifests rather than false, so the schema cannot distinguish "not positional" from "not yet assessed".

   5. Tracker consistency

   Check	Verdict
   18 repointed, 0 landed	confirmed
   Five repoints name #826 and #55, dated 2026-08-07	confirmed-with-caveat
   CATALOG.md current	confirmed
   build_audit.py scan --strict exits 0	confirmed
   Orphans 26, down from 31	confirmed-with-caveat
   lectures/datasets/ holds exactly three files	confirmed

   Parsed rather than grepped: 18 records, all repointed, zero landed. The asserted grep trap is real — the unanchored form returns 19, the 19th match being the comment at migration.yml:341. CATALOG.md regenerates byte-identical. The strict scan exits 0, and it is not vacuous: re-run against the pre-session tree it correctly emits nom_balances.npy: marked repointed but consumers still read via [local-path…]. The three surviving files in lectures/datasets/ were confirmed four ways, including --ignored to catch strays.

   Orphans: 26 now, 31 before, and the delta is the five files by name, not merely by count. The caveat is that 31 is the intermediate state. The true pre-session baseline was 32 — wasm#55 also deleted six already-orphaned mirrors — so the real movement was 32 → 31 → 26.

   The date caveat: both PRs merged 2026-08-06T23:46Z and 23:49Z UTC. 2026-08-07 is the local AEST date. migration.yml states no timezone convention.

   6. Blind spots

   Check	Verdict
   Confirm the ws#25 regression, and the three sync PRs	confirmed-with-caveat
   The cause: sweep was nine repos, described as org-wide	confirmed
   Re-run the sweep properly, org-wide	see below — this is a finding
   Six prose references repointed; no build would have caught them	confirmed / confirmed-with-caveat
   These were the last local-path data reads	confirmed-with-caveat
   (comment) Restored line live, :263 reads name_chinese	confirmed-with-caveat
   (comment) No check detects an overwritten localisation	confirmed-with-caveat
   (comment) The collision is intermittent	confirmed-with-caveat
   (comment) Sweep remaining tracks for future collisions	confirmed-with-caveat

   The cause is confirmed, and PR #827's wording is worse than this issue says. Verbatim from #827: "An org-wide sweep over every repo's origin/main finds no remaining reader of these five paths. The only matches are data-lectures' own sidecar manifests and its audit annotations, neither of which is a read." Not just the label — the conclusion was false at the moment it was written. #827 merged at 01:35:54Z; zh-cn's main was then 0466562, which read three of the five paths, and was not superseded until 17 minutes later.

   The properly-scoped sweep: 253 non-archived repos enumerated, 252 shallow-cloned (only QuantEcon/.github is unclonable), all grepped. Clone-and-grep rather than code search, which does not index URL strings reliably.

   Within live lecture series and their downstream dependents, lecture-intro.zh-cn was the only reader, and it is fixed. The .notebooks mirrors (intro, python, programming, advanced — all pushed within the last week) are clean; lecture-python-intro.notebooks already carries the data-lectures URLs. The other translations never read these files. lecture-datascience.myst reads none of them.

   Two repos outside that boundary still resolve the deleted paths, and they are executable reads, not prose links — reported below rather than buried here.

   The zh-cn caveats are all overcounts in the issue's own wording. "Eight datasets" is eight lines resolving to six distinct files; caron.npy and nom_balances.npy were read from zh-cn's own committed copies by local path and never 404'd. "Nine call sites depend on name_chinese" is really one — :263 — with six more passing code_to_name into that helper. long_run_growth.md:141 and :263 are exact, no drift, and the CSV's 27 rows resolve every code the lecture uses; the reverted alternative reproduces KeyError: 'name_chinese' first-hand. "#286 and #288 touched URL lines only" is materially wrong for #288, which also rewrote two np.load calls into requests+BytesIO and added imports — though the load-bearing point (neither overwrote a localisation) stands.

   "No check anywhere detects an overwritten localisation" is false as worded, and worse in substance: a purpose-built deterministic guard does exist in action-translation (src/cli/target-local-reads.ts), written three weeks earlier for this exact country_code_cn.csv / name_chinese case — but it is wired only into the CLI's forward command and is absent from the action bundle CI runs (findTargetLocalReads, verifyPreservedReads, buildPreserveInstruction all grep to 0 hits in dist-action/index.js). Nothing protected this sync because the protection was never shipped to the path that needed it.

   The prose check confirms cleanly — 6 references, 3 per repo, lines 60-62, all now 200, verified from git blobs and from both published sites. "No build would have caught them" was demonstrated rather than argued: a jupyter-book built with intro's exact -nW --keep-going flags and a deliberately-404 link reports nothing, and myst build --html likewise without --check-links, which wasm's CI never passes. One correction — intro does have linkcheck.yml, but it is cron-only, fail-on-broken: false, and runs over the release tarball, so it files an issue rather than failing a build.

   "Last local-path data reads" is confirmed for datasets — 0 local-path reads in either repo, and the detector is live (it still reports 4 for lecture-python-advanced.myst). Strictly there is one remaining local open(): short_path.md:463 reads graph.txt, which the same lecture writes at :272 via %%file. The audit classifies this as embedded, correctly.

   7. Decisions settled

   Check	Verdict
   storage_options timeout silently ignored	confirmed
   Publish failures were upstream, not configuration	confirmed-with-caveat
   The high_dim_data branch story is false	refuted

   The pandas claim is confirmed against pandas 3.0.5 itself, and proved rather than read: io/common.py:403-404 is verbatim as quoted, a local echo server receives Timeout: 5 and X-Probe: hello as literal HTTP headers, and against a server that accepts and never replies the read blocks past 25s identically with {"timeout": 5}, {"timeout": "5"} and None — no timeout effect, no warning for the unknown key. QuantEcon.manual#149 rests on solid ground.

   Both halves of the publish claim verify — the diff is empty (confirmed by blob identity across all three refs, not just diff text), and setup-miniconda@v4 resolves to v4.0.1's SHA in all four log bundles, released 2026-04-24, before the last green publish. But "upstream, not configuration" misnames the surface. Both failures died fetching https://conda.anaconda.org/default/linux-64/repodata.json — channel default, singular, which is what environment.yml literally declares. That is not conda's defaults alias; it is a real but empty anaconda.org user channel returning {"packages":{}} in 160 bytes. defaults — served from repo.anaconda.com — succeeded in every attempt, with ToS accepted mid-failure. All 20 timeout errors in attempt 1 name conda.anaconda.org; none name repo.anaconda.com.

   Section 7's third item is refuted. The first conjunct holds (delete_branch_on_merge: false, so the deletion was manual). The second is backwards: the Events API records DeleteEvent ref=update_scf_noweights at 2026-07-15T23:57:24Z, 12 seconds after the merge — and the four repoints merged 19 to 24 minutes later (00:16:19Z, 00:16:22Z, 00:20:17Z, 00:21:07Z). So the branch was gone before the consumers were repointed, not after. The ws#14 correction's conclusions — "no consumer ever 404'd" and "the ordering rule was followed here, not violated" — are the reverse of the timeline.

   8. Deliberately not done

   Check	Verdict
   fig_3.ods kept; intro's is the last copy in the org	refuted
   Copilot's request declined, not overlooked	confirmed
   environment.yml untouched, recorded as a deliberate scope decision	refuted

   The Copilot item is clean and well-evidenced: the request landed 23:23:08Z, the reply declining it 23:42:41Z with three stated reasons, and the merge 23:46:08Z — a decision predating the merge by 3m27s, not an oversight. The corpus claim it rests on re-derives exactly: requests.get 8 hits, raise_for_status 0 across all nine clones.

   "The last copy in the org" is false. A Trees-API scan of all 277 org repos on their default branches finds blob 5d6ae11c on four non-archived repos: lecture-python-intro, lecture-intro.zh-cn (pushed 2026-08-07), test-actions-lecture-intro and python-lecture-sandpit.myst. The other three sub-claims hold — wasm's was a redundant unreferenced mirror, data-lectures holds none, and the format-twin evidence re-measures correctly (both parse to one Sheet1, DataFrame.equals true) — but the one the retention decision rests on does not.

   "Confirm this was recorded" fails. environment.yml genuinely is unchanged — blob-identical across publish-2026jul17, aug06, aug06b, aug07 and main. But no record of the scope decision exists outside this issue. A sweep of every issue and PR comment since 2026-08-06 across lecture-python-intro, lecture-wasm, data-lectures, workspace-lectures, QuantEcon.manual and meta (~250 KB) returns zero hits for conda/environment.yml/channel; #826 and #827 never mention publish or conda. The only written trace is #51 — the artifact under validation, written by the same agent.

   ---

   Findings

   Filed separately rather than buried here, per the brief. Everything below survived an adversarial pass; the corrections that pass forced are already applied.

   Inside the lecture repos:

   Finding	Where
   channels: - default (singular, empty channel) in seven workspace repos — the endpoint both failed publish attempts died on	lecture-python-intro + 6 siblings
   Piplite index 404s, so %pip renders a red error block as the first visible output on all 14 wasm lectures that have one	lecture-wasm
   /myst-theme.css requested without BASE_URL, 404 on every page (10/10 sampled)	lecture-wasm
   Three pandas FutureWarnings printed to readers ('M'→'ME', 'AS'→'YS') — already fixed upstream in intro	lecture-wasm
   Data Sources documents 3 of the 5 datasets the lecture reads	both french_rev.md
   A stray 30-row DataFrame renders above fr_fig3 on the published page	lecture-python-intro
   fig_3.ods retention rests on a false premise (4 org copies, not 1)	lecture-python-intro
   The ws#14 correction is itself wrong — branch deleted before the repoints	workspace-lectures#14
   assignat.xlsx.yml:97 says 14 unread sheets, lists 15	data-lectures
   build_audit.py committed_referenced() basename fallback is unreachable	data-lectures

   Translation programme (auto-synced, so out of this workspace's hand-sweep, but the guard gap is why ws#25 happened): the target-local-read guard is not compiled into the action bundle and not wired into sync mode; it also cannot see a target-local read expressed as a URL, which is how lecture-python.zh-cn's country_map.csv is written. rebase-translations.yml has been dead in lecture-intro.zh-cn since 2026-07-21 — QUANTECON_SERVICES_PAT's selected-repository list does not include it. zh-cn still commits eight orphaned dataset copies. The forward sweep found 15 collision rows, 4 high, the sharpest being heavy_tails.md:862, where the read cell also holds country_names = ['美国','日本','印度','意大利'] consumed at :881 — and P3's media→raw rewrite must edit exactly that line.

   Outside live lecture series and their dependents — reported for triage, no action taken:

   QuantEcon/tom-econ370-2025 sets base_url to the deleted intro path at lectures/french_rev.md:70-75 and drives 13 read_excel calls from it; french_rev is in its _toc.yml, Pages is live, and the published page serves three 404 links today. Last push 2025-02-04. QuantEcon/test-actions-lecture-intro has the same reads plus long_run_growth.md:89, but its _toc.yml was trimmed to five canary lectures on 2026-08-05 with only_build_toc_files: true, so they are latent — they go red the moment the full book is restored, which is the false-alarm mode that trim exists to prevent.

   Ten non-archived repos carry lecture content, have not been pushed in over 12 months, and all ten have Pages enabled. Separately, 24 repos outside the workspace's nine read data by URL from a QuantEcon lecture repo — the standing blast radius of every future repoint. The .notebooks mirrors and translations self-heal; the course and workshop repos do not, and nothing watches them.

   Checkboxes left for the issue owner.
3. mmcky commented on Aug 7, 2026

   

   mmcky

   on Aug 7, 2026

   ContributorAuthor

   More actions

   Validation complete, and every finding is now routed. Closing.

   33 of 37 checkboxes ticked. The four left open are not outstanding work — each has a settled disposition:

   Left open	Why	Where it went
   §1 Wayback pixel comparison	Not-completable by construction. The Wayback Machine holds french_rev.html but has never archived a single one of its figure PNGs — CDX empty for all 15, id_ fetches 404 on 15 of 15. The newest pre-migration capture is also 2026-03-09, predating two commits that changed the figure set, so it would have been a contaminated baseline regardless.	Substantive claim re-derived from the pre- and post-migration build artifacts: identical 18-file _images list, all 18 sha256-identical. Pixel-identity confirmed.
   §7 high_dim_data auto-delete story	Refuted. Branch deleted 12s after merge; the four repoints merged 19-24 min later.	QuantEcon/workspace-lectures#14
   §8 fig_3.ods last copy in the org	Refuted. Blob 5d6ae11c on four non-archived repos.	QuantEcon/workspace-lectures#14
   §8 environment.yml recorded as deliberate	Refuted. Unchanged, yes; recorded nowhere but this issue.	QuantEcon/lecture-python-programming#607

   Where the findings went

   Routing index at QuantEcon/workspace-lectures#27. In short: #607 widened from 3 editions to 7 repos with the publish-failure evidence attached; the 15-row translation collision table posted to QuantEcon/workspace-lectures#26, which it answers; five more archive candidates added to QuantEcon/workspace-lectures#18; piplite severity evidence to QuantEcon/lecture-wasm#42; the PAT grant gap to QuantEcon/meta#319; and five new issues — QuantEcon/action-translation#253, QuantEcon/lecture-wasm#56, QuantEcon/lecture-wasm#57, QuantEcon/lecture-python-intro#828, #52.

   What this exercise was worth

   Two things justify the format, and one qualifies it.

   The named bias was the right thing to name. Re-running the checks by different methods — git object store, a real browser executing Pyodide, build artifacts, local socket servers, the Trees API — is what produced all three refutations. None of them would have surfaced by re-running the original session's fetch-and-compare.

   The sweep scope mattered more than the sweep. #827's claim was not merely mislabelled "org-wide"; its conclusion was false at the moment it was written. Re-running it properly over all 253 non-archived repos found two further repos that still resolve the deleted paths. The lesson for future sweeps is to scope explicitly to live lecture series plus downstream dependents, and to clone-and-grep rather than trust gh search code, which cannot find URLs.

   The qualification: the adversarial pass discarded 21 of 45 candidate findings, and corrected two that survived — including one of this validation's own claims that rested on a 27s latency measurement which turned out to be 0.3s. A validation run without that pass would have shipped a materially wrong finding into a public issue. Worth keeping in the template.
4. mmcky

   closed this as completedon Aug 7, 2026