promote: a refresh with --exclude erases the excluded series' provenance, so the ledger follows whoever ran last and a later stale copy is refused #19

Follow-up from the review of #14.

What happens

A refresh re-resolves each target entry over the series the run considers, then writes the entry whole. In resolve_sources, sources and divergent come from the run's eligible series and superseded from its considered series (tools/promote:815-818). The ledger write rebuilds the entry and its promoted_at pins from those lists (tools/promote:1487-1514), and asset_provenance rebuilds each asset's sources the same way (tools/promote:1465-1483). So a run with --exclude deletes every sources, divergent and superseded row of the excluded series, with its pin, from each entry it rewrites, although it never looked at those copies. A by-name re-promotion with --exclude does the same.

CI always refreshes with --exclude $PROMOTE_EXCLUDE, set to dp jax (.github/workflows/drift-check.yml:71, used at :142). Since #14, a local run excludes nothing by default: ALWAYS_EXCLUDED is gone, and dp, now a consumer, is recorded under superseded. So the ledger's provenance depends on who ran last. A default local run (by name, --toc or a bare --refresh) writes superseded: dp rows, jax rows and their pins. The next nightly refresh of those entries deletes them again. It reports an entry whose text did not move as unchanged, so the change shows only in the refresh PR's ledger diff.

The deletion also changes behaviour. Since #14 (7128fb1), a differing copy is spared as stale only while it is recorded under sources (tools/promote:718-726). Once a CI refresh has erased that record:

• drift-check stops watching the copy, because it compares only the copies recorded in the entry (tools/drift-check:362). There is no stale-copy line, and a later edit to that copy raises no diverged finding.
• The next run that considers the series refuses a merely stale copy as "differs across series ... reconcile or rename it upstream, or name the canonical series in sync/canonical.yml". In CI that is a red nightly. Following the advice would record the stale copy as divergent and mark the entry interim.

Excluding dp protects nothing

dp is a consumer, so it can never collide:

• resolve_sources compares only canonical-eligible copies (tools/promote:657), and records a consumer's copy under superseded whatever it holds (:818).
• The asset checks, check_admonition_identity and asset_provenance, also iterate only eligible series (:1380, :1474).
• refresh_targets treats a consumer-canonical entry the same way whether or not the consumer is excluded: it skips the entry while its pin stands and fails it once the pin moves (:1099-1105).

Considering dp in the refresh therefore cannot fail an entry or change a canonical choice; excluding it only strips its superseded rows. On the committed ledger, with the mirror at its pins, tools/promote --refresh --exclude jax --dry-run plans the same 41 targets, with 0 failures, as --exclude dp jax --dry-run. The only difference is superseded=['dp'] on 39 of them. The refresh job already reconstructs every series (drift-check.yml:132), so mirror/dp is there.

The comment above PROMOTE_EXCLUDE (drift-check.yml:60-70) still gives the reason from before dp became a consumer: the real dp series "is excluded while the dp-test sandbox stands in for it", and leaves the list at QuantEcon/project-monorepo#24.

Reproduction (with the tools/tests harness)

CI_REFRESH = ("--refresh", "--exclude", "dp", "jax")  # PROMOTE_EXCLUDE in drift-check.yml

# 1. A consumer's superseded row and pin follow whoever ran last.
repo = make_repo(["intermediate", "dp", "jax"], consumers=["dp"])
repo.lecture("intermediate", "career", lecture_text("Career"))
repo.lecture("dp", "career", lecture_text("Career (dp's older copy)"))
assert repo.promote("career").returncode == 0            # a default run: superseded [dp], dp pinned
repo.move_pin("intermediate")                            # any intermediate commit
res = repo.promote(*CI_REFRESH)
assert "summary [refresh]: 0 promoted, 1 unchanged" in res.stdout
assert "superseded" not in repo.entry("career")          # dp's row is gone ...
assert "dp" not in repo.entry("career")["promoted_at"]   # ... with its pin
assert repo.promote("career").returncode == 0            # the next default run
assert "superseded" in repo.entry("career")              # puts them back

# 2. An erased source turns a later stale copy into a collision.
repo = make_repo(["programming", "dp", "jax"], consumers=["dp"])
for s in ("programming", "jax"):
    repo.lecture(s, "autodiff", lecture_text("Autodiff"))  # identical in both
assert repo.promote("autodiff").returncode == 0          # a default run: sources [programming, jax]
repo.move_pin("programming")                             # any programming commit
assert repo.promote(*CI_REFRESH).returncode == 0         # now sources [programming], no jax pin
repo.lecture("programming", "autodiff", lecture_text("Autodiff", "A fix."))  # jax has not ported it
repo.move_pin("programming", 3)
assert "[stale-copy]" not in repo.drift_check().stdout   # jax's copy is no longer watched
res = repo.promote("--refresh")                          # jax considered again
assert res.returncode == 1                               # the stale copy is refused

Output of the last refresh:

summary [refresh]: 0 promoted, 0 unchanged, 0 skipped, 1 failed
[autodiff] FAILED: autodiff.md differs across series (same: programming:e49607d9b6bd; different: jax:db61c0bcf04e) — a same-named file must be one lecture: reconcile or rename it upstream, or name the canonical series in sync/canonical.yml

db61c0bcf04e is jax's copy exactly as it was promoted. Without the CI refresh in between, the same history gives a different result. drift-check reports [stale-copy] lectures/autodiff.md: the jax copy is unchanged while programming moved — jax is now stale, and the refresh exits 0 with stale: jax at an earlier promoted version.

When it bites

• Today: nothing is lost yet. The committed ledger has no superseded item and no dp or jax row, so CI's refresh has nothing to erase. The churn starts with the first default local run that touches a lecture dp or jax also holds. For example, a bare tools/promote --refresh on the committed state would add superseded: dp to 38 entries and jax's divergent row to ifp_egm. It would also refuse short_path, as intended while jax's renames are pending. CI's next refresh of those entries takes the rows out again.
• superseded rows: README.md:103 says a consumer's copies are recorded as superseded. Under CI's flags they stay recorded only until the entry's next refresh. When the dp switch-over (QuantEcon/project-monorepo#24) records dp as a consumer, for example with tools/promote --toc dp, every entry dp holds gains superseded: dp. While PROMOTE_EXCLUDE lists dp, each nightly refresh strips those rows from the entries of whichever series moved. drift-check never reads superseded, so this is ledger churn only.
• Refusals: these need a canonical-eligible series that is recorded under sources and later left out of a run: jax in CI, or any series a local run passes to --exclude. The first realistic case is autodiff, once the planned merge (QuantEcon/project-monorepo#76) makes it identical in programming and jax. A default run records jax as a source, and the next nightly refresh after programming's pin moves erases that record. Once jax leaves PROMOTE_EXCLUDE, a jax copy one fix behind programming is refused, and the nightly stays red until jax ports the fix.
• On main: excluded rows were already dropped. dp was an ordinary series, which CI excluded and local runs considered, so its sources/divergent rows came and went in the same way. jax was always excluded, so it never had rows. promote: differing same-name copies are an error, jax is considered, and a consumer series class (project-monorepo#74) #14 brings jax into the churn, and makes the erasure matter: on main, a differing copy became divergent and the refresh stayed green.

Suggested fix

1. In a refresh, and in a by-name re-promotion, carry each excluded series' recorded rows forward unchanged. That covers its sources, divergent and superseded items (digest included), its promoted_at pin, and its rows in the asset records that asset_provenance rebuilds. The stale rule then still sees the record when the series is considered again, and drift-check keeps watching the copy. A throwaway patch along these lines at the ledger write (tools/promote:1487-1514) makes both reproductions above fail, keeps the existing suite green, and lets scenario 2's later refresh keep jax as stale. The alternative is to refuse to rewrite an entry that records an excluded series, naming that series. That would turn the nightly red instead, so carrying the rows forward looks better.
2. Drop dp from PROMOTE_EXCLUDE now, and correct the comment above it. jax is then the only entry; when it leaves, once its renames land, the list is empty, which the refresh step handles since promote: differing same-name copies are an error, jax is considered, and a consumer series class (project-monorepo#74) #14 (it leaves the flag off, drift-check.yml:142).
3. Add both scenarios to tools/tests, asserting the corrected behaviour.

Line references are at #14's head as it merges, with #17 included.