Skip to content

business_cycle: read four data-lectures snapshots and re-enable the lecture - #85

Draft
mmcky wants to merge 1 commit into
mainfrom
wasm/business-cycle-snapshots
Draft

mmcky wants to merge 1 commit into
mainfrom
wasm/business-cycle-snapshots

Conversation

@mmcky

@mmcky mmcky commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Re-enables business_cycle on this site. The lecture was left out because its data reads were live API calls, wbgapi against the World Bank and pandas_datareader against FRED. It now reads four cached snapshots that QuantEcon/data-lectures serves and refreshes by pull request, so it can run in the reader's browser. This is block B of the wasm-dynamic-snapshots project, tracked in QuantEcon/data-lectures#118.

Closes #70.

What changes

  • lectures/business_cycle.md is a port of lecture-python-intro main's copy, not an edit of this repo's 2025-10-23 copy. The port is from intro commit 901462c; the file last changed there in Update business cycle data to latest available (#364) lecture-python-intro#770. So it carries intro's later changes and its "to the present" prose, which suits snapshots that refresh. That is option (a) of decision 1 on lecture-wasm dynamic snapshots: business_cycle reads the four data-lectures snapshots (P4) data-lectures#118, taken on 2026-09-30. This repo's conventions from ENH: Migrate to use mystmd and quantecon-theme.  #31 are kept: :label: options with :::{figure} blocks, short-form tags and the front matter. Beyond those conventions, only data access changes:
    • the pip install cell and both API imports go, and pyodide_http.patch_all() runs in the imports cell, as in mle and simple_linear_regression;
    • each wb.data.DataFrame(…, labels=True) becomes pd.read_csv(…, index_col='economy'), followed by .loc[[…]] on the economies the API call named;
    • each web.DataReader(id, 'fred', start, end) becomes fred_series(id, start, end), a helper described below;
    • the two wbgapi teaching calls, wb.series.info and wb.series.metadata.get, become non-executing python blocks, with a note linking the intro edition that runs them live;
    • the two year-on-year changes pass fill_method=None to pct_change, described below.
  • lectures/myst.yml uncomments business_cycle, between long_run_growth and inflation_history.
  • README.md: business_cycle and short_path leave the unsupported list, so the list matches the six entries still commented out in myst.yml (short_path has been enabled in myst.yml since ENH: Migrate to use mystmd and quantecon-theme.  #31). The rule "do not edit lectures here; sync from intro's wasm branch" gives way to "edit lectures/ directly and check the change on the PR's preview". That branch has not changed since 2025-04-24 (c870cca) and is 131 commits behind intro main, and PRs here already edit lectures/ directly.

Enabling the page also mends a link. On main, long_run_growth's [business cycle](business_cycle) builds as a download of the raw .md file; on this branch it links to the page. Nothing is deleted, because this repo holds no copy of these data.

The four snapshots

All four are read at main on raw.githubusercontent.com, the form that works in the browser (data-lectures repoint rule 5).

URL Source Read at
https://raw.githubusercontent.com/QuantEcon/data-lectures/main/lectures/gdp_growth_annual.csv World Bank NY.GDP.MKTP.KD.ZG, nine economies lines 78, 102 and 503
https://raw.githubusercontent.com/QuantEcon/data-lectures/main/lectures/unemployment_rate_annual.csv World Bank SL.UEM.TOTL.NE.ZS, four economies line 568
https://raw.githubusercontent.com/QuantEcon/data-lectures/main/lectures/private_credit_to_gdp.csv World Bank FS.AST.PRVT.GD.ZS, United Kingdom line 746
https://raw.githubusercontent.com/QuantEcon/data-lectures/main/lectures/us_business_cycle_monthly.csv FRED M0892AUSM156SNBR, UNRATE, USREC, UMCSENT, CPILFESL and INDPRO on one monthly index line 319, then fred_series at eight call sites

For the lecture's authors to accept

The fred_series helper

This is the one piece of new logic that intro does not have (lines 322 to 328):

def fred_series(series_id, start, end):
    """
    One series from the snapshot, between start and end, without the
    empty months before its first observation or after its last.
    """
    data = fred.loc[start:end, [series_id]]
    return data.loc[data.first_valid_index():data.last_valid_index()]

The composite pads all six series to one index starting in 1919, but web.DataReader returned only a series' own observations in the window. The helper returns the same thing. I compared it live with web.DataReader (pandas-datareader 0.11.1) at all eight call sites on 2026-09-30. The frames have the same columns, index name and dtype, value dtypes and null placement, including UNRATE's 2025-10 hole, which FRED also serves. Values differ only where FRED has revised INDPRO since the snapshot was taken, by at most 0.13 in the YoY change, and by FRED's newer month.

The alternative is to repeat the slice and the trim at each of the eight call sites. The helper has one limit, which no call here reaches: for a window with no observations it returns every row of the window, all NaN, rather than an empty frame.

fill_method=None in the two pct_change calls (lines 645 and 704)

Pyodide 0.27.0, which this site loads, ships pandas 2.2.3. There, pct_change(12) pads FRED's 2025-10 hole in CPILFESL forward. That plots a point intro's figure does not have and prints a FutureWarning under the consumption figure. Intro builds on pandas 3 (anaconda 2026.07), where fill_method=None is already the behaviour, so intro does neither. Passing it explicitly makes the browser's figure match intro's, and pandas 3 accepts it unchanged. INDPRO has no hole, so the production figure does not change; the keyword is there for symmetry. Drop both if you would rather keep intro's code verbatim, at the cost of the padded point and the warning.

For an author's eye: five sentences

Two sentences are new, in Data acquisition:

  • line 59: "This version of the lecture runs in your browser and reads cached snapshots of the World Bank and FRED data from QuantEcon/data-lectures." (linked to that repo's catalog)
  • line 61: "The two calls to the World Bank's data API wbgapi below are shown without running; the edition of this lecture on intro.quantecon.org runs them live." (linked to intro's Data acquisition section)

Three existing sentences are reworded, because the port made them untrue:

  • line 47: "We pull all time series through to the most recent available data." now reads "We pull all time series through to the most recent data in the snapshots we read below." A snapshot is refreshed by pull request, so it can trail its source by a month.
  • line 74: "Now we use this series ID to obtain the data." now reads "Now we read a snapshot of this series." The code reads a file, not the ID.
  • line 85: "We can look at the series' metadata to learn more about the series (click to expand)." now ends at "series.", because the block no longer runs and has no hidden output to expand.

The first non-executing block also gains import wbgapi as wb, because wb is otherwise never bound on the page. The code comment at line 101 now reads "Read the same snapshot again".

What was verified, and how (2026-09-30)

Check How Result
Every code cell runs All 22 {code-cell} blocks of the committed file, in order, in one namespace, under the Pyodide 0.27.0 pins that this build's thebe bundle loads (pandas 2.2.3, numpy 2.0.2, matplotlib 3.8.4; CPython 3.12.13, where Pyodide has 3.12.7). Agg backend, a no-op pyodide_http stub, and imports of wbgapi and pandas_datareader blocked, since the browser has neither No errors and no warnings. 12 figures, all with data. The only network reads are the four URLs above
The site builds npx mystmd@1.9.1 build --html in lectures/, as ci.yml does, on this branch rebased on main (ff9196d, theme v3.2.0) Exit 0. The only new message is the "Extension inferred for table of contents entry" notice that every toc entry gets. The page has 22 executable cells, the two static blocks as plain code, and 12 numbered figures (us_gdp to dcpc)
data-url-guard Its run: step, taken from .github/workflows/data-url-guard.yml and run on this branch Passes
URL form Every URL in a code cell, plus data-lectures' own audit scan (scripts/build_audit.py at main) applied to the file Six reads of four files, all https://raw.githubusercontent.com/QuantEcon/data-lectures/main/lectures/<file>, with no github.com/…/raw/ form, no media host and no other ref. The audit finds four data-lectures refs and no live-API marker; on main it flags wbgapi and pandas_datareader
The URLs serve curl with Origin: https://quantecon.github.io, and again with a Netlify preview origin Each returns HTTP 200 with access-control-allow-origin: *, with bytes matching its manifest's sha256 on data-lectures main at the time
World Bank reads Live wb.data.DataFrame(…, labels=True) at all four call sites, against the snapshot reads Same columns, Country labels, nulls and values; only the row order differs
fred_series The live comparison above As described above
No conflict with #81 git merge-tree against #81's head Merges cleanly

Not verified: a run in a browser. The code ran on CPython with pyodide_http stubbed, so the browser's fetch path is untested here.

The gate: Run all on this PR's preview

Someone needs to open this PR's own preview at https://pr-<N>--beautiful-dodol-cb9543.netlify.app/business-cycle/, where N is this PR's number (the "Netlify Preview Ready!" comment gives the URL), and run "Run all". JupyterLite stops at the first failing cell, so the lecture passes when all 12 figures draw. Ignore the red Netlify app check: it fails on every PR here (#62), while the workflow's own preview serves real pages. CI cannot do this check, because ci.yml builds without running any code. Run it after data-lectures' first monthly refresh has merged (its PR should open on Monday 2026-10-05), so the gate tests the bytes this page will read.

Before merging

Open questions

  • Should update_lectures.py go? This PR keeps it. The README now retires the sync, but the script still downloads intro's wasm branch and copies its lectures/ over this repo's (shutil.copytree(…, dirs_exist_ok=True)). That branch still has a business_cycle.md, so running the script would bring back the old live-API copy, along with other stale files. Delete the script, or keep it as history? Without it, porting from intro means applying its conversions by hand (!pip to %pip, dropping --upgrade).
  • Could the two wbgapi calls run here after all? Include business_cycle: what it reads, what data-lectures now serves, and the sequence #70 assumed the API hosts send no CORS headers. Today api.worldbank.org/v2 answers access-control-allow-origin: *, though FRED's fredgraph.csv sends none. Whether wbgapi installs through piplite and runs under pyodide_http is untested, so this PR follows Include business_cycle: what it reads, what data-lectures now serves, and the sequence #70 and shows the calls without running them. Its prose does not claim they cannot run.
  • Michigan's citation on the page. data-lectures rests UMCSENT's licence on Michigan's FAQ, which asks users to cite "University of Michigan, Survey Research Center, Surveys of Consumers". The Consumption subsection names "the University of Michigan" and links FRED, as intro does. Adding Michigan's form of the citation there is the authors' call.
  • Row order, cosmetic: the two GDP tables list the economies in the lecture's order (USA, ARG, GBR, GRC, JPN), while live wbgapi returns the API's order. Values, labels and figures are unaffected.

Left alone on purpose

This is prose inherited from intro that the snapshots do not fully bear out. Changing it is intro's call, not this port's.

  • "from 1970 to the present" for UK credit: the series starts in 1960, at 17.34% of GDP.
  • Argentina's growth "did not fall during the two developed economy recessions in the 1970s and 1990s": it is -0.03% in 1975, the last year of the shaded 1973 to 1975 band.
  • The comment above end_date, "To freeze the data for a reproducible build…": this site builds without executing code, and a fixed date does not freeze a snapshot that is refreshed in place.

🤖 Generated with Claude Code

…ecture

The lecture has been left out of this site because its data reads are
live API calls: wbgapi against the World Bank and pandas_datareader
against FRED. It now reads four cached snapshots from
QuantEcon/data-lectures on raw.githubusercontent.com at main, the form
that serves access-control-allow-origin: * (repoint rule 5):

  gdp_growth_annual.csv          World Bank NY.GDP.MKTP.KD.ZG
  unemployment_rate_annual.csv   World Bank SL.UEM.TOTL.NE.ZS
  private_credit_to_gdp.csv      World Bank FS.AST.PRVT.GD.ZS
  us_business_cycle_monthly.csv  FRED M0892AUSM156SNBR, UNRATE, USREC,
                                 UMCSENT, CPILFESL and INDPRO

The file is a port of lecture-python-intro main's business_cycle.md
(901462c; the file last changed there in
QuantEcon/lecture-python-intro#770), not an edit of this repo's
2025-10-23 copy. So it carries intro's later changes and its "to the
present" prose, which fits snapshots that refresh (FRED monthly, the
World Bank annually). This repo's conventions from #31 are kept: the
:label: options with :::{figure} blocks, short-form tags, front matter.

Beyond those conventions, only data access changes:

- the pip cell and both API imports go; pyodide_http.patch_all() runs
  in the imports cell, as in mle and simple_linear_regression;
- each wb.data.DataFrame(..., labels=True) becomes a read_csv with
  index_col='economy', followed by .loc on the economies the API call
  named;
- each web.DataReader(id, 'fred', start, end) becomes
  fred_series(id, start, end), a small helper over the composite that
  returns what DataReader returned: one column named for the series,
  holding only that series' own observations in the window;
- the two wbgapi teaching calls, wb.series.info and
  wb.series.metadata.get, are shown as non-executing python blocks,
  with a note linking the intro edition that runs them live;
- the two year-on-year changes pass fill_method=None to pct_change.
  Pyodide 0.27.0 ships pandas 2.2.3, which otherwise pads FRED's
  2025-10 hole in CPILFESL and warns; intro builds on pandas 3, which
  does neither, so the browser's figure now matches intro's.

Two sentences are new (the page reads snapshots; the API calls are
shown without running), and three are reworded because the port made
them untrue: "through to the most recent available data", "use this
series ID to obtain the data" and the metadata sentence's "(click to
expand)".

myst.yml uncomments business_cycle. The README drops business_cycle
and short_path from the unsupported list (short_path has been enabled
in myst.yml since #31), and retires the rule to sync from intro's wasm
branch: that branch has not changed since 2025-04-24, and PRs here edit
lectures/ directly. update_lectures.py stays for now.

Verified on 2026-09-30:
- all 22 code cells run in order in one namespace under the Pyodide
  0.27.0 pins (Python 3.12, pandas 2.2.3, numpy 2.0.2, matplotlib
  3.8.4; pyodide_http stubbed, wbgapi and pandas_datareader blocked),
  drawing 12 figures with no errors and no warnings;
- mystmd 1.9.1 builds the site (exit 0), and its only business_cycle
  message is the toc-extension notice every entry gets;
- the data-url-guard step passes;
- all four URLs return 200 with access-control-allow-origin: *, and
  their bytes match the manifests' sha256;
- the World Bank reads match live wbgapi at all four call sites (only
  the row order differs), and fred_series matches web.DataReader at
  all eight FRED call sites in columns, index, dtypes and nulls; values
  differ only by FRED's latest INDPRO revisions and its newer month.
Not verified: a run in a browser.

Block B of QuantEcon/data-lectures#118; the issue here is #70.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@netlify

netlify Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

❌ Deploy Preview for beautiful-dodol-cb9543 failed.

Name Link
🔨 Latest commit adaf9bc
🔍 Latest deploy log https://app.netlify.com/projects/beautiful-dodol-cb9543/deploys/6abc9634f1e0fe0008690634

@github-actions

Copy link
Copy Markdown

📖 Netlify Preview Ready!

Preview URL: https://pr-85--beautiful-dodol-cb9543.netlify.app (adaf9bc)

✨ Browse the preview at the URL above.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Include business_cycle: what it reads, what data-lectures now serves, and the sequence

1 participant