pandas_datareader is maintained again (0.11.0/0.11.1, June 2026) — reconsider it as the builder-side entry point for dynamic data #26

Activity

1. mmcky

   added

   questionFurther information is requestedFurther information is requested

   on Jul 29, 2026
2. mmcky commented on Jul 29, 2026

   

   mmcky

   on Jul 29, 2026

   ContributorAuthor

   More actions

   Correction to the inventory above. The table originally said seven lectures across three repos. It is ten across four — the first pass searched for pandas_datareader (underscore) and missed the lectures that write it hyphenated in their !pip install line. The body has been updated; the three additions are networks and input_output in lecture-python-intro, and doubts_or_variability in lecture-python-advanced.myst, a repo previously reported as clean.

   The correction sharpens rather than weakens the case. Four of the ten never import the package at all — they only install it. doubts_or_variability is the clearest case: a comment in that lecture records that it deliberately avoids pandas-datareader in favour of FRED's CSV endpoint, so the migration already happened there and only the install line survives. Those four lines can be removed with no code change and no dependency on the questions below.
3. mmcky commented on Aug 6, 2026

   

   mmcky

   on Aug 6, 2026

   ContributorAuthor

   More actions

   Recommendation: split this into three questions — one is free today, one is a clear yes, one is a "not yet"

   Re-verified the inventory against current origin/main of all four repos today (2026-08-06). The three vestigial installs are all still there — networks and input_output in lecture-python-intro, doubts_or_variability in lecture-python-advanced.myst each carry a pip install pandas-datareader line with no matching import, while the five real users (business_cycle, unemployment_linear, unemployment_shocks, phillips_lost_conquest, phillips_two_stories) have both.

   The three questions at the bottom of this issue have different answers, and bundling them is what makes this feel undecidable.

   1. Delete the vestigial installs — do this now, independent of everything else

   Three lecture pages install a package they never import. doubts_or_variability is the instructive one: it carries an in-lecture comment recording that it deliberately reads FRED's CSV endpoint instead, so the migration already happened and only the install line survived. Removing these three lines is a no-op for output, shrinks three build environments, and needs no decision from the questions below. It should not wait for P4.

   2. Move pandas_datareader out of lecture runtime — yes, unambiguously

   This is the part PLAN.md Phase 5 already commits to and this issue frames correctly: a fetch that runs on a cadence in scripts/, whose failures surface in one canary rather than in seven lecture repos' CI, and lectures read a stable URL. Nothing here depends on which client the builder uses. AGENTS.md states the principle already — live API calls are for teaching data access, not for getting data.

   3. Make pandas_datareader.macro the standard fetch layer — not yet, and probably not for UNRATE

   Recommending no to standardising, for three reasons.

   The four-stage contract is already the standard, and it is the right level. fetch() returns raw, preprocess() is pure, validate() reads the manifest. That interface is what makes builders uniform and swappable; mandating a specific client underneath adds no uniformity the contract does not already provide, and a fetch stage is typically five lines. Standardising the library rather than the interface buys little and constrains a lot.

   The maintenance history argues for keeping the blast radius small. The package was effectively unmaintained from July 2021 to June 2026. Two releases in one week in June 2026 is genuinely good news and worth acting on — but it is not yet a track record. Making it the mandatory entry point for every dynamic dataset re-creates the single-point fragility Phase 5 exists to remove, just relocated from seven repos into one chokepoint. QuantEcon/lecture-python.myst#924 is the reminder of what that costs when it breaks.

   For UNRATE specifically there is a lower-risk option with in-repo precedent. A cron builder fetching one FRED series needs a plain HTTP GET against fredgraph.csv — no dependency, no wrapper, no version surface. That is exactly the migration doubts_or_variability already made on purpose, and lecture-python-programming's pandas/polars lessons already teach the same endpoint. Writing P4 against read_macro would be adding a dependency to replace a URL.

   Where pandas_datareader.macro would earn its place: a builder that genuinely needs many series across FRED and Eurostat and OECD with one normalised frame. No current or planned dataset does — the FRED incidental set in the migration plan's Wave 7 is FRED-only. Worth revisiting the moment a multi-source builder is actually on the table, and this issue is the right place to keep that open.

   4. Does the live path stay in lectures?

   Keep it exactly where the fetch is the lesson — lecture-python-programming's pandas and polars — and nowhere else. Those two are annotated pedagogy: lesson in the audit for precisely this reason and should keep calling live APIs, because the call is the content. For the fifteen incidental and mixed lectures, the target is snapshot-only with the live call shown but not executed, which is the switch mechanism P4 has to document.

   Effect on QuantEcon/meta#160

   That issue's premise — "last released 2021, maintainer not inclined to maintain the readers" — is now factually out of date and it should be re-read against this before anyone acts on it. But the recommendation above does not resurrect the package as a lecture dependency either: items 1 and 2 still reduce its footprint substantially. The two issues are compatible.

   Related: #14 (the builder contract these fetch stages implement) and the P4 work in PLAN.md Phase 8.
4. mmcky commented on Sep 1, 2026

   

   mmcky

   on Sep 1, 2026

   ContributorAuthor

   More actions

   Decided and acted on, 2026-09-01 — all three questions from the recommendation above.

   1. The vestigial installs — done. lecture-python-intro dropped networks and input_output's lines in QuantEcon/lecture-python-intro#803; the last one, doubts_or_variability, is QuantEcon/lecture-python-advanced.myst#381.

   2. pandas_datareader out of lecture runtime — yes, and the mechanism now exists. The refresh-as-PR workflow and weekly canary landed in #110; the first FRED snapshots built on it are in #114.

   3. Not pandas_datareader.macro — a shared library of our own, and one builder per source. #114 adds builders/_fred.py, a Fred class over fredgraph.csv (fred.series('UNRATE'), fred.frame([...]), server-side frequency change, the observation_date → DATE rename), so a FRED builder's fetch stage is one line and its validate() is the only thing worth reading. The first user is builders/business_cycle_fred.py, which writes the intro business_cycle lecture's six FRED series as one composite monthly file — one file per source rather than one per series, because six URLs in a lecture is what makes it hard to read, and the composite's columns are the lecture's own variable names. Testing stays obvious: the class is testable alone, and each builder's contract is a single validate().

   4. The live path stays where the fetch is the lesson — lecture-python-programming's pandas and polars, and now explicitly lecture-python-intro's business_cycle (decided today: intro keeps every live call; lecture-wasm, which cannot make them, reads the snapshots — QuantEcon/lecture-wasm#70).

   pandas_datareader.macro remains worth revisiting the day a builder genuinely needs FRED and Eurostat and OECD in one frame; none does yet. Leaving this open only for that — say the word and I'll close it as decided.
5. mmcky

   added a commit that references this issue on Sep 1, 2026

   P4: the business_cycle snapshot set — three World Bank tables and a F…

   ...

   ebb1bd1