Skip to content

fix(release): retry the post-publish npm check, and verify all 69 packages instead of 1 - #15331

Open
hotlong wants to merge 2 commits into
mainfrom
claude/issue-15321-publish-verification
Open

fix(release): retry the post-publish npm check, and verify all 69 packages instead of 1#15331
hotlong wants to merge 2 commits into
mainfrom
claude/issue-15321-publish-verification

Conversation

@hotlong

@hotlong hotlong commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Fixes #15321

The last step of the publish job verified the release with one npm view of
@objectstack/cli. On the 17.3.0 release (run 8330, job 100996109699) it exited
1 on a release that had succeeded completely. Two defects, both fixed here.

① It raced the registry. npm committed @objectstack/cli@17.3.0 at
10:53:25.108Z and the check read at 10:53:32 — seven seconds later — and got an
absence. npm's write path and its CDN-fronted read path are eventually
consistent, and one shot immediately after a 69-package burst bets the release
on how fast reads settle.

② It looked at 1 package of 69. Had a package genuinely failed to publish,
the step could not have seen it — and the false red on cli would have masked
it. Measured on the same release, @objectstack/runtime did not commit until
11:00:32, so between 10:53 and 11:00 the fixed group really was partially
readable and nothing was looking.

What changed

.github/workflows/release.yml — the publish job's last step now calls
node scripts/release-verify-npm.mjs in place of the inline npm view. Same
step, same id: publish, same published=true output, same if ! …; then exit 1; fi shape. $VERSION is handed over as RELEASE_VERSION, the spelling the
two sibling steps already use.

Nothing about publish GATING is touched. environment: release, the
publish-pending predicate and its load-bearing success() && (...)
parenthesisation, force's dispatch-only semantics, the on: block and the
whole release-integrity job are byte-identical. This change is entirely about
what happens after a publish, never whether one runs. git diff on that
file is one step body.

scripts/release-verify-npm.mjs — new. Bounded backoff, whole derived package
set, failure names what is absent. It also writes a short receipt to
$GITHUB_STEP_SUMMARY saying what was checked and how long the read path took
to settle, so the next person to tune the budget has measurements rather than a
docblock.

.github/workflows/lint.yml — runs the new script's --self-test, which
check:self-test-wired requires and which is the only instrument on this logic.

⛔ It still fails CLOSED

The retry absorbs latency, never absence. A package still missing when the
budget is spent exits non-zero, named; so does a registry that could not be read
at all ("we could not tell" is not "it is there"). There is no || true, no
downgrade to a warning, and no path that reports success without having read
every target. The vacuity direction is guarded too: an empty target set is a
hard failure, not a green run of zero checks.

Where the retry budget comes from

Not a guess — npm's own time field on that release. The spread from the first
read (cli, 10:53:25, which is when verification starts) to the last commit
(runtime, 11:00:32) is 7m07s. The budget is 15 minutes: a little over
twice the observed spread, because burst latency scales with whatever else the
registry is absorbing. Backoff is 5s doubling to a 60s cap, with the final wait
truncated so the schedule lands on the budget exactly. A self-test case pins
RETRY_BUDGET_MS > 7m07s, so shrinking the budget below the measurement reds.

Cost in the other direction: a genuinely broken publish is reported 15 minutes
later — paid once, by a job that has already spent longer than that building.

Why the package set is derived from the workspace

Every non-private workspace package, each at the version its own manifest
declares — 69 today, and --dry-run prints them. Membership comes from
listWorkspacePackages in scripts/release-github-releases.mjs, the release
lane's existing answer to "which packages are publishable", itself built on
scripts/workspace-enumerator.mjs — this repo's one parse of
pnpm-workspace.yaml. Reusing it rather than writing a fourth private copy of
manifest.private !== true is the point: two release-lane scripts disagreeing
about which packages a release contains is its own defect.

Why this source rather than changeset publish's stdout — the direction each
fails in.
Parsing the publisher's own report is self-referential: if the
publish died early, printed nothing, or changed its output format, the parse
yields an empty set and the verification passes having checked nothing. That
is the silent-success direction, and it is the class of bug this card is about.
The workspace derivation is independent of the publish's belief — it is the set
that ought to be on npm, computed from the tree the guard step already proved
matches github.sha.

What happens if it misses a package. It cannot under-select against
changeset publish, whose own publishable set is the same workspace filter: a
package outside the workspace is not something this repo can publish at all. The
one way it could over-select is a future non-empty .changeset/config.json
ignore (empty today) — and that direction is loud: the run goes red naming
that package, noticed within one release, rather than passing in silence. Two
existing gates keep the derivation honest and neither is this one:
check-changeset-fixed.mjs (run by this same job, before the build) asserts
every non-private workspace package is in the Changesets fixed group, and a
package this repo does not publish is marked private: true in its own
manifest, which is exactly the field the filter reads.

The version is read per package rather than assumed from one release number.
The fixed group makes those equal today; this is a derivation that does not
depend on it.

The failure message

The old one named @objectstack/cli on a release where cli was the one
package definitely published, so whoever picked it up went and looked at a
healthy package. The new one names only what is absent, says how many of how
many, distinguishes "not on the registry" from "registry unreadable", and points
at the repair channel that actually exists:

::error::publish ran but 1 of 69 package(s) are still not readable on npm after 19 round(s) over 900s: @objectstack/runtime@17.3.0

Not readable after the full retry budget (900s, 19 rounds):
  - @objectstack/runtime@17.3.0 — not on the registry

The other 68 package(s) ARE readable, so this is a partial publish, not a failed one.
Repair: re-run Release via workflow_dispatch with `force: true`. `changeset publish` skips
versions the registry already has, so it republishes only what is listed above — a repair,
never a duplicate.

Reproduced, then shown green

The regression needs no npm. The self-test drives the real code with a stub
registry whose read path settles — a package that is committed but answers
absent for its first N reads — and expresses the old shape as parameters of
the same function (targets: [cli], budgetMs: 0), so the old behaviour is
pinned by this code rather than by a second implementation nothing holds to it.

reading shape result
① the false red one cold read on cli, one target, no retry RED — names @objectstack/cli, the package that was fine; registry read exactly once
① absorbed same registry, 3 targets, 15m budget GREEN — 3 rounds of backoff, all three checked
② the masked defect runtime never lands, one target, no retry GREEN — never even asked about @objectstack/runtime
② caught same registry, 3 targets RED — names @objectstack/runtime and nothing else

52 cases across 9 batteries. The self-test carries the #13489 registered-case
floor: a battery that stops running reds by set difference, and the roster's own
size is pinned — a verifier whose own cases can silently stop running is the same
class of bug as the one being fixed.

Verification

At 1ba59a8ed. Every exit code captured before any pipe
(cmd > log 2>&1; EXIT=$?), and every verdict quoted from the line the tool
itself printed.

  • node scripts/release-verify-npm.mjs --self-testexit 0OK release-verify-npm self-test: 52 cases pass across 9 batteries
  • node scripts/release-verify-npm.mjs --dry-runexit 069 package(s) would be verified on npm, all at 17.3.0
  • live probe against registry.npmjs.org: @objectstack/cli@17.3.0true,
    @objectstack/cli@99.99.99false. The probe is exercised against the real
    registry, not only against the stub.
  • pnpm exec eslint --no-inline-config scripts/release-verify-npm.mjs
    exit 0, 0 errors 0 warnings

Four ablations, each run from the committed state, each mutation proven on
disk by counting the removed anchor and the injected marker, each restore
proven by git hash-object matching the HEAD blob with git diff HEAD empty:

ablation mutated restored
battery 3 stops registering (total unchanged) exit 1battery "3. The masked defect…" DID NOT RUN — 0 cases registered, 6 pinned exit 0, all cases
RETRY_BUDGET_MS → 0 exit 1, 5 cases — incl. NEW SHAPE absorbs the cold read now rounds=1 waits=0 exit 0, all cases
derivation collapses to cli only --dry-run1 package(s) would be verified 69 package(s) would be verified
job summary written via a deferred promise exit 1ENOENT … summary.md on the immediate read-back exit 0, all cases

The battery-floor ablation is the one that matters for the #13799 recipe: it
leaves the total unchanged while battery 3 registers 0, which is exactly the
regression a pinned total cannot see and a registered-name floor can.

Two ablation attempts were discarded as void readings, not re-read until
they went green — both because the proof that the mutation landed did not hold,
which means the ablation did not run:

  • the derivation ablation, first attempt: its anchor was a two-line pattern
    handed to grep -c -F, which treats a newline as a pattern separator and
    counted 59 matching lines. Redone with a single-line anchor.
  • the summary ablation, first attempt: a perl -0pi whose escaping failed to
    compile, so the file was untouched — the anchor count stayed 1 and the marker
    count 0, and the run was refused rather than read. Redone with a node edit.

Recorded because a silent re-run until something lands is the same defect one
layer up.

Gates. node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands derived 45 commands at 1ba59a8ed; all 45 run. 43 exit 0. The
two non-zero are both self-classified as non-findings by the gates themselves
and are reported as NOT MEASURED, never as passes:

  • check-required-contexts.mjs --verify-required-setexit 2, required-set sweep: NOT VERIFIED — … HTTP 401. The gate's own words: "NOT VERIFIED is not a
    pass and not a failure of the tree; exit 2 classifies the ENVIRONMENT." This
    container has no GitHub token. Retried under NODE_OPTIONS=--use-env-proxy as
    the gate suggests: still exit 2 (403). Unmet prerequisite: a token.
    pnpm check:required-contexts (the tree half) exits 0.
  • pnpm check:type-check-debtexit 3, PREREQUISITE NOT MET — 33 workspace dependenc(ies) … have no built type entry point on disk. Exit 3 is that
    gate's distinct "nothing was measured" code. Unmet prerequisite: the built
    dependency closure. This diff contains no TypeScript, so no package's
    type-check surface moves. pnpm check:type-check-coverage exits 0.

Lint narrowing, declared. pnpm lint scans the whole repo and CI runs it
once regardless; the local run is narrowed to the changed file, and the
narrowing is measured rather than assumed. (i) The two .yml files are outside
eslint's population; the one .mjs file is inside it — eslint linted it rather
than reporting it ignored. (ii) --format json reports exactly 1 file
linted, 0 errors, 0 warnings. (iii) No parserOptions.project and no
projectService appears anywhere in eslint.config.mjs (line 328 states the
absence outright), so type-aware linting is off and every file's verdict is a
function of that file plus the config — neither of which this diff moves for any
untouched file. The narrowing therefore excludes nothing.

The out-of-scope claim is machine-checked, not asserted. In
release.yml, everything from the top of the file through the line before the
edited step body is byte-identical to eb40a7210, and everything from
- name: Create GitHub Releases to EOF (50 lines) is byte-identical. The diff
removes exactly three lines, all three inside the verification block. The
success() && (...), release-integrity:, workflow_dispatch: and
refresh_version_pr occurrence counts are unchanged; environment: release and
publish-pending each gain exactly one occurrence, both on one line — the new
comment saying the gating is not this check's business.

No changeset — skip-changeset

.github/workflows/** and scripts/** publish nothing from any package, so
this is the skip-changeset label case, not an empty changeset (rejected by
#5471, and #4898 is that failure silently blocking a publish).

⛔ No auto-merge is armed. release.yml is the one file in the repository that
publishes; a human reviews it.


🤖 Generated with Claude Code

https://claude.ai/code/session_01UHvF5hyiZjnCyExFnfQB8m

Generated by Claude Code


Generated by Claude Code

github-actions Bot and others added 2 commits September 4, 2026 11:37
…ages

The last step of the publish job verified the release with one `npm view` of
@objectstack/cli. On 17.3.0 it exited 1 on a release that had succeeded
completely.

It raced the registry: npm committed cli@17.3.0 at 10:53:25.108 and the check
read at 10:53:32, seven seconds later, and got an absence. The write path and
the CDN-fronted read path are eventually consistent.

And it looked at 1 package of 69, so a package that genuinely failed to publish
is the one case it could not see — and the false red on cli would have masked
it. @objectstack/runtime did not commit until 11:00:32.

scripts/release-verify-npm.mjs replaces it: bounded backoff over the whole
derived publishable set, with the absent packages named on failure. It still
fails closed — the retry absorbs latency, never absence — and an empty target
set is refused rather than passed vacuously.

Publish gating is untouched: everything up to the step body and everything from
"Create GitHub Releases" to EOF is byte-identical.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UHvF5hyiZjnCyExFnfQB8m
The append was `import('node:fs').then(...)` — a floating promise, discarded by
the caller's `process.exit(await main())`. It wrote the summary on neither path
while reading, in review, exactly like one that did.

Battery 9 pins it the only way this class can be pinned: append, then read the
file back with no await in between. A deferred write fails that case.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UHvF5hyiZjnCyExFnfQB8m
@hotlong hotlong added the skip-changeset PR has no user-facing published change; bypasses the changeset gate label Sep 4, 2026 — with Claude
@github-actions github-actions Bot added the ci/cd label Sep 4, 2026

hotlong commented Sep 4, 2026

Copy link
Copy Markdown
Contributor Author

复核:PASS

⚠️ 同会话自审声明:本复核由派发本卡的同一 PM 席位做出,与 dev 席同会话,不是独立第二意见。下列「已核」均为我自己在 head 1ba59a8ed 上跑的 diff 层面读数。

这张卡我只有一件事必须自己验:发布门控有没有被动

因为这是仓库里唯一发布 npm 的文件。已核,没动:

  • release.yml 的整个 diff 只删了三行,全在校验块内(一行注释 + 那次 npm view + 它的 ::error::);
  • environment: release(:928)、success() && (...) 及其承重括号(:949-951)、publish-pending 谓词、force 的 dispatch-only 语义 —— 全部不在 diff 里;
  • 步骤保留了 id、published=true 输出,以及 if ! ...; then exit 1; fi 的形状,所以没有任何 if: 谓词或 job-output 接线移动。

fail closed 成立,逐字:

if ! RELEASE_VERSION="$VERSION" node scripts/release-verify-npm.mjs; then
  exit 1
fi
echo "published=true" >> "$GITHUB_OUTPUT"

新代码里那段注释也把边界写死了 ——「⚠️ This checks what happens AFTER a publish, never WHETHER one runs」—— 这句该留着。

空目标集的拒绝:已核,而且是这次交付里最值钱的一处

if (!Array.isArray(targets) || targets.length === 0) {
  problems.push(
    'the publishable-package derivation produced NO targets. Verifying zero packages would report '
      + '"every published package is on npm" having read nothing — refusing to pass vacuously.',
  );

⭐ 这正是一小时前我在 #15261 上抓到的同一类口子(账本清零后 rows.length >= 0 恒真,控制变平凡)。差别是:那次要我指出来,这次席位自己堵上了,还给了它一整个 battery(「6. Targets are derived, and an empty derivation is refused」,下限 7 例)。

自测台账本身也按 #13489 的正确形状做的 —— 钉的是 battery 的名字而不是数字,并且台账自身的大小也被钉住(SELF_TEST_BATTERY_FLOOR = 9),因为「删掉一条和把它归零一样有效地让那条下限失声」。头部注释把因果说穿了:

failures.length === 0 as a self-test's only success condition makes "every case held" and "the cases never ran" print the same line — which is the same class of defect as the one this script fixes

15 分钟预算是测出来的,不是拍的

取自 17.3.0 那次发布自己的 npm time 字段:最后一个包(@objectstack/runtime)比第一次读取晚 7分07秒。预算按实测的两倍留余量,而不是「感觉半小时够」。

三处工程纪律值得点名

  1. 它在自己的初稿里抓到一个真 bug 并单独成 commit:GITHUB_STEP_SUMMARY 的 append 是个浮动 promise,被 process.exit 丢弃 —— 两条路径上都没写成,而读起来跟写成了一模一样。改成同步,并由 battery 9「读回文件、中间不 await」钉住。这个失败形状(看起来在做、其实没做)与本卡要修的缺陷同源。
  2. 两次消融尝试被判 VOID 并重做,而不是重读到绿:[c] 用了两行锚点配 grep -c -F,而 -F 把换行当模式分隔符,数出 59 行 —— 落盘证明不成立;[d] 的 perl -0pi 转义没编译,文件根本没被改(锚点计数仍 1、marker 仍 0)。两次都拒绝采信。
  3. 两条门禁明确记为 NOT MEASURED 而非通过:check-required-contexts --verify-required-set exit 2(容器无 GitHub token,按网关建议加 --use-env-proxy 重试仍 403 —— 门禁自己的话是「exit 2 classifies the ENVIRONMENT, not the tree」);check:type-check-debt exit 3(缺 33 个依赖的构建产物),并补了一句真正的论据:本 diff 不含 TypeScript,类型检查面没有移动,且 check:type-check-coverage exit 0。

一处汇报遗漏(不影响判定)

报告的文件清单没列 .github/workflows/lint.yml(+24 行)。我核过了,那是把 --self-test 接进门禁工作流 —— 合理,而且是 check-self-test-wired 门禁要求的,注释也写清了为什么这个自测比一般的更要紧(「the production path runs once per release, on a runner, with npm's real timing, so nothing else can tell a working retry loop from a broken one」)。但改了哪些文件应当在报告里全列,下次补上。

落地

我不武装 auto-merge,这条不改。 release.yml 是仓库唯一发布 npm 的文件,由人复核 —— 与 Governed Surface Guard 怎么判那条路径无关,我不赌它的清单。

skip-changeset 席位已自行打上并回读(工作流与 scripts/ 都不在已发布包里,不是「写一个空 changeset」的情形)。


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/cd size/xl skip-changeset PR has no user-facing published change; bypasses the changeset gate

Projects

None yet

1 participant