feat(service-automation): an operator can put back a suspension a failed resume consumed - #13951
Conversation
📓 Docs Drift CheckThis PR changes 2 package(s): 5 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:
⛔ 2 release-owned page(s) also name something this change touched. These are read-only:
What this run could not see
Coarse fallback — 6 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): Which tree this was computed onThis run read A worktree cut from an older # while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 34b191f34ce9b8cf0cc822dfcbc5dbb989fc1dc0 && git checkout 34b191f34ce9b8cf0cc822dfcbc5dbb989fc1dc0
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 836a29c27aa9dbac27cc643106e5b2bf68cc96c1 f422d353a47903e24282f415bce247084a3c7e64 && git checkout -B drift-repro 836a29c27aa9dbac27cc643106e5b2bf68cc96c1 && git merge --no-ff f422d353a47903e24282f415bce247084a3c7e64
node scripts/docs-audit/affected-docs.mjs --json 836a29c27aa9dbac27cc643106e5b2bf68cc96c1
|
⚖️ 契约复审裁决 — FAIL。逐字采纳,⛔ 标签不清,交回返工
核验读数(采信的前提)派发席的处置(⛔ 与裁决分开,裁决在上,原文未动)
⭐ 值得单说:Finding 2 抓的是「这个 PR 自己证明它知道这一类过期」 —— 它在 Generated by Claude Code |
… and true up the sys_automation_run declaration Contract-review rework (ruling recorded on PR 13951; behaviour unchanged): - index.ts: export type SuspensionRestoreResult / SuspensionRestoreRefusal in the engine.js type block. The verb was already barrel-reachable, so the eight refusal values were published de facto while the union was unnameable de jure — no exhaustive switch, no annotated result, no typed handler parameter. ConsumedSuspension deliberately stays unexported (it appears in no barrel-reachable signature). - consumed-suspension-restore.test.ts: barrel-import pin. The existing suite imports from './engine.js', which is why the gap had no witness. The pin annotates a real result with the BARREL type and writes the exhaustive switch (never-typed default), so removing the export breaks tsc; the runtime half pins the verb on the class the barrel itself exports. - sys-automation-run.object.ts: the declaration asserted invariants this PR falsified. One-class corrections only — the node_type description and the trigger-attribution comment now carve out the consumed-suspension class of terminal row (the same correction shape this PR already applied in plugin-approvals' backfill), and variables_json's description names the presence-discriminator the store's deserializer keys off. - engine.ts: the NO_CONSUMED_SUSPENSION docblock no longer claims "exists and is terminal" — the arm also answers on a non-terminal logged status, and the reason string was already honest about that. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ZC5rNQj3WEet5HAmmAkMs
…ntime strings check:doc-authoring is right: the two descriptions the rework corrected carried '#13909' inside runtime string prose, which reaches operators who cannot resolve a tracker id. The carve-out text stays; the ids move to adjacent comments (the reader who can resolve them reads the source). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ZC5rNQj3WEet5HAmmAkMs
契约复审(返工复核):PASS —— 逐字采纳,双载体标签已清转录核验(采信前必做): ⇒ 每一轮都在档、零回退 ⇒ 裁决合法,取逐字采纳。以下为复审席原文,⛔ 未改写、未删节、未润色: 契约复审(返工复核):PASS结论一行两项阻塞均已实缴:桶导出与经桶见证成立,对象声明三处与读写代码逐一相符,准予发布。 非阻塞
我核了什么 / 我没核什么核了(全部读
没核(NOT MEASURED,及归属):
PM 侧收口
非阻塞项 1 的处置:不再开一轮返工⛔ 不为一行注释的点号路径再走一趟「返工 + 再复审」—— 那一轮的成本远超收益,而复审席自己判它 ⇒ 按仓内既有惯例记成待搭车项: Generated by Claude Code |
Part of #13909 — slice 2, the operator exit verb (the parent's deliverable 2, "shape 4").
Slice 1 (PR #13934) made the condition visible and measured that it is terminal. This slice adds the way out. It does not decide the resume ordering: that is #13937, unruled and in the maintainer's hands, and this branch leaves the ordering,
forgetSuspendedRunandtraverseNextexactly as they are.Tier — declared, not self-cleared
node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstackprinted, against this diff:⇒ the clause-② PATH limb does not fire: nothing under
packages/specis touched. The derivation's next line is the one I am not entitled to answer:This change does widen a public surface — one new public method on
AutomationEngine. Whether that makes it clause-② by the content limb is the PM's judgment, not mine, and I am not clearing it. ⛔ Please rule before merge.packages/spec: it was placed where its nearest sibling already lives, not moved to dodge a review.Why the verb is an engine method and not a spec contract method
The dispatch warned a new engine verb "very likely lands in
packages/spec/src/contracts/automation-service.ts" and told me not to contort the design to avoid that. I did not. I placed it beside its exact sibling, and here is the reading:IAutomationService(inpackages/spec) declaresexecute,resume,listRuns,getRun,listSuspendedRuns,getSuspendedScreenand friends — the flow-execution surface a transport calls.cancelRunis not in it. The nearest sibling of this verb — a deliberate, operator/service-initiated action on a suspension's lifecycle, outside the ordinary execute/resume path — isAutomationEngine.cancelRun, and it is engine-level.restoreConsumedSuspensioniscancelRun's mirror image (one consumes a suspension on purpose, the other puts one back on purpose), so it sits in the same place.AutomationResult.codewould put eight new codes into a contract every transport reads — and would mint platform vocabulary for a condition [Decision] Workflow resume ordering: a thrown node today leaves the run terminally unresumable — which of three shapes, given that the current order buys exactly-once across a crash? #13937 has not yet ruled the shape of.⛔ No platform-level name or status is minted.
AutomationResult.statusis still'completed' | 'paused' | 'failed', the run staysfailed, and nothing here names the condition. Naming is an explicit same-batch sub-item of #13937 (under shape 2 the condition largely stops existing).Consequence, stated rather than hidden: like
cancelRun, this verb has no REST route and no CLI command in this PR. A host holding the engine can call it; an operator reaches it through a host. Adding a door needs a permission model of its own and is filed separately.What it does
AutomationEngine.restoreConsumedSuspension(runId, { requestedBy, reason })puts back the suspension a resume consumed, so the run is resumable again through an ordinaryresume— same authority gate, same screen validation, sameresumingguard.Deliberate, never automatic. Nothing calls it. No retry, no sweeper, no self-healing arm: an operator asks for one run, by id.
Verbatim, never advanced. What goes back is the pause as it stood — its own variables, its step log as of the pause, its node, node type, correlation, screen. Two consequences, both in the docblock, the trace and the return value:
signal.branchLabelis not part of a suspension at all, so a silent replay would take a different edge;Restoring the pause exactly as it stood is also what keeps this from pre-empting #13937: it is precisely the state a resume-ordering change would have left behind, so it composes with that ruling instead of racing it.
Where the snapshot lives. In-memory hot journal (bounded, oldest-first) plus — when a store is configured — the run's own terminal history row, in
sys_automation_runcolumns that already exist (variables_json/context_json/screen_json/correlation/node_type) and that the object's own field comment already complained were never written on terminal rows. No new column, no new status value, no new table, no new store method. A run that is restored and then finishes upserts that row without a snapshot, so "restorable" cannot outlive the condition. Over a byte budget the snapshot is dropped, never truncated — half a variable map would restore a run into a state it was never in, and the drop is logged with the size.The durable half is not a nicety: ADR-0019 exists because the process hibernates between suspend and resume, and these runs are found hours later by a sweep or a support ticket, in some other process. An exit that only worked inside the lifetime that stranded the run would answer almost never.
Refusals — the reasons are the deliverable
Each is earned by a specific observation, in this precedence:
RESTORE_IN_PROGRESSRESUME_IN_PROGRESSresumingholds the id — a resume is in flight, outcome not decidedRUN_SUSPENDEDSTORE_UNAVAILABLERUN_COMPLETEDcompletedRUN_CANCELLEDcancelled(ADR-0044)NO_CONSUMED_SUSPENSIONRUN_NOT_FOUND⭐
pausedsplits into two. "Suspension gone, no terminal row yet" is what a live resume looks like from outside, and re-arming one of those races it — its traversal would finish and recordcompletedwhile a paused row it knows nothing about survives.RESUME_IN_PROGRESSis that case;RUN_SUSPENDEDis the ordinary already-parked one.⛔
STORE_UNAVAILABLEis a refusal on purpose. Reading an outage as "no suspension" is the one mistake that would put a second resumable pause on a live run, so the live-suspension probe uses the STRICT loader, and the history read refuses rather than degrading to "nothing to restore" — an answer an operator would act on by giving up.⛔
NO_CONSUMED_SUSPENSIONdeliberately does not claim WHICH of its two causes it is. Nothing in the engine can tell "never suspended" from "snapshot no longer held"; the message names the status actually observed instead of inventing a certainty.Idempotence — carried by the paused row, not by a flag
A suspension is keyed by run id, so however many operators ask there is one resumable pause: the second call finds one live and answers
RUN_SUSPENDED. That holds across processes and across a restart, because the paused row is durable. Arestoringset adds the in-process half, exactly asresumingdoes for resume, so two callers racing in one process get onerestored: trueand one refusal rather than two claims.And it cannot produce two traversals by construction: the verb does not resume. It re-arms the pause and stops.
The trace
The whole reason this class stayed silent is that nothing recorded it, so an exit that is itself invisible repeats the defect. A restore writes:
warnnaming the run, flow and node, with who asked, why, when the suspension was consumed and the original failure in the structured slot (all three are uncontrolled text — the finding(service-automation): engine.ts 还剩三处同形的 warn message 拼接 —— forgetSuspendedRun / cancelRun / listSuspendedRunsDurable,是 #5912+#6230 之后该文件的最后一批 #6299 family — and a newline in any of them would split one record into several physical lines of which only the first is greppable).warn, notinfo, because this moves a run the platform had recorded as terminally failed. When nobody suppliedrequestedBythe record saysnot recorded, which is itself a finding;pausedrun-log entry, so the repair is not invisible on the surfaces that read run status. This is the same record the resume path writes when a run re-suspends, for the same reason.failed, becausegetRun/listRunsdeliberately let a terminal row win over a paused one. A durable status that survives the restart is naming work, and naming is #13937's same-batch sub-item.Disclosure boundary — measured
sys_automation_run.variables_jsonnow appears on one class of terminal row. Who may read it is unchanged: #7900 converged both doors on thesys_automation_runobject-read grant, status-independent, and the maintainer's 2026-08-12 ruling rejected per-field filtering as the mechanism. What changes is which rows carry the snapshot, behind that same grant.⛔ The run-detail HTTP surface is untouched for failed runs: the snapshot is a parameter to
recordLog, deliberately not a field ofExecutionLogEntry, whichGET /automation/:name/runs/:runIdserves verbatim — that would have been the disclosure widening #7639 refused. The onlyvariablesa caller newly sees there belong to a run that IS paused, which is #7639's own sanctioned case.In-place correction in
plugin-approvals(comment only, named here because it is not this card's package)backfill-platform-row-organizations.tsdocumentedcontext_jsonas "Written on paused rows only —recordTerminaldoes not persist it". This change makes that false for one class of row, so the comment is corrected in place. No behaviour change, and none is needed: a row that has the snapshot uses it exactly as a paused row does, which is strictly better for that sweep (a subject whose live record is gone now has a write-time snapshot where it had none). Scan for other readers of those columns:runtime/src/domains/automation.ts(the permission gate above), thesys_automation_runobject definition, and dogfood assertions about paused rows — none of them branch on "terminal rows have no snapshot".Verification
All at final commit
4c949ca39, working tree clean. Exit codes captured before any pipe (cmd > file 2>&1; EXIT=$?); verdicts quoted from each gate's own printed line.Suites (via
scripts/pm/os-verify-lock.sh):pnpm --filter @objectstack/service-automation test—Test Files 92 passed (92),Tests 1116 passed (1116), VERDICT command-exit 0. The new file alone: 22 passed.RunRecordandrecordLogchanged and the comment fix lands in approvals:pnpm --filter @objectstack/plugin-approvals --filter @objectstack/runtime test— approvalsTests 632 passed (632), runtimeTests 3011 passed (3011), VERDICT command-exit 0.pnpm --filter '@objectstack/service-automation^...' build, VERDICT command-exit 0.Type coverage, measured rather than assumed. This package has no
typecheckscript —pnpm --filter ... typecheckanswersERR_PNPM_RECURSIVE_RUN_NO_SCRIPTand exits 1, loudly rather than silently. So it was measured directly:tsc --noEmit -p tsconfig.jsonreports exactly 3 errors, all pre-existing, all innested-region-parity.test.ts(Property 'flows' is private), a file this diff does not touch and whose text is identical onorigin/main.--listFilesconfirms the new test file is in the program (1 hit each for the test file,engine.tsandsuspended-run-store.ts) — unlikeplugin-approvals, whose tsconfig excludes**/*.test.ts. The ratchet that owns those 3:check:type-check-debt --re-measure—OK — 29 ledger entr(ies) re-measured in 209.0s, 1531 raw tsc error(s) total, none above its recorded number/surplus: none — every entry sits exactly at its measurement, so any new error is red.Gates. Re-derived from the ACTUAL diff with
node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack(no paths passed — the script takes the change set from the merge base), both output sections read whole: 30 path-matched families plus the convention-triggered sets for a test-file edit, for a package owning ani18n-extract.config.ts, and for a file carrying an ADR-0112 code-shaped value. 36 of 37 harvested commands GREEN, including every ratchet:check:engine-double-contract(no new fake engine — this PR adds a test file with none),check:where-matcher(321 matcher(s) ... 0 silently-wrong),check:query-options-erasure,check:cross-package-test-inputs,check:published-files,check:test-source-alias,check:type-check-coverage(66/78 workspace packages type-checked),check:i18n(9 package(s) — all bundles in sync),check:i18n-stale-fill,check:dual-build-cjs-loads,check:nul-bytes, and — the one worth naming because a new refusal vocabulary is exactly its population —check:dispatcher-error-vocabulary, green. Plus a manualgrep -naP '[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]'over all five changed files: 0 hits.pnpm lint(repo-wideeslint . --no-inline-config) run whole under the lock, VERDICT command-exit 0 — so there is no narrowing to declare.ONE gate NOT MEASURED, by its own printed verdict — neither a pass nor a red:
check-test-completeness.mjsexit 3,PREREQUISITE NOT MET — this gate grades a saved 'turbo run test' log, and no log was named. CI tees that log; a local run cannot produce it.check:type-check-debtfirst answered exit 1 with a build prerequisite, not a red:--re-measure cannot run: 1 workspace dependenc(ies) ... have a type entry point OLDER than their own sources(the ablation legs had movedengine.ts's mtime after the closure build). That closure was cleared exactly as the gate named it (pnpm --filter @objectstack/service-automation build) and the gate re-run green, rather than reported as a failure.Ablation — two legs, direction predicted before each run, both matched exactly
The implementation was committed first (
4c949ca39), then mutated in the working tree.No rebuild is in the resolution path, and here is why: the test imports
./engine.jsand./suspended-run-store.jsrelatively, from the same package and directory, so vitest resolves them tosrc, never through a packageexportsmap todist. The colour changes below are themselves the corroboration — a stale artefact cannot redden ten named tests and then green them again.Mutation confirmed ON DISK, never by an editor's exit code. HEAD blob
883fc7fb00cfd31157467d0f402738d04983888b. Both mutation scripts refuse unless every anchor matches exactly once, both abort if the mutated blob equals the HEAD blob, and both count the injected marker AND the deleted text on disk. (Leg B's first attempt aborted on that count — the deleted-text pattern also matchedresumeInternal's ownRESUME_IN_PROGRESSguard, so the anchor was tightened and re-run rather than waved through.)6d66db03a2ec7ef3c15ac4455da8090b528b0ef6, injected marker 2, deleted text 0. Predicted: the ten restore-dependent pins redden; every refusal pin stays green, because none of them depends on a snapshot existing. Observed:Tests 10 failed | 12 passed (22)— the ten being the three exit pins, both idempotence-restore pins, both trace pins, and the three across-a-restart pins.346ec81e3d069839df7375d8e627abd22c332241, injected marker 1, deleted text 0. Predicted: exactly one pin reddens, the in-flight refusal. Observed:Tests 1 failed | 21 passed (22),expected 'NO_CONSUMED_SUSPENSION' to be 'RESUME_IN_PROGRESS'.⭐ What leg B actually measured, reported honestly rather than overclaimed: the guard's contribution here is the reason, not the outcome. With it removed, a restore attempted mid-traversal still cannot re-arm anything, because the journal entry is written in the catch arm and does not yet exist — the durable design makes that instant structurally unreachable. The guard is what makes the engine say the true thing instead of "nothing to restore".⚠️ There is a narrower window it does cover for real — between the journal write and
resuming'sfinally— which is guarded but not pinned; pinning it needs a seam inside the catch arm that this slice does not add.DECLARED CONTROLS — green in BOTH directions, reported as controls and NOT as ablation evidence: under A, all eight refusal pins, "does not itself traverse" (with no journal the restores refuse, so the traversal count is 1 either way), and the three "what this slice deliberately does NOT do" pins. Under B, everything except the one named pin.
RESTORE PROVEN BY STATE after each leg, not by an exit code:
git diff HEADempty,git hash-objectback to883fc7fb..., residual ablation markers 0. The harness carriestrap restore EXIT INT TERMwith absolute paths resolved fromgit rev-parse --show-toplevel, and restores withgit checkout HEAD -- ABSOLUTE_PATH— never a baregit checkout --, which reads from the index the mutation itself wrote.⭐ The pins earned their keep before any ablation. An early revision of this branch silently dropped the
await this.forgetSuspendedRun(run, 'resumed')call while inserting the step-count capture beside it — i.e. it accidentally implemented #13937 shape 2, the very thing this slice must not pre-empt.leaves the resume ordering exactly as it isis the pin that caught it, and it is in the file for that reason.NOT MEASURED — needs the deployment, not this repo
⛔ How many runs are in this state in any real deployment cannot be read from this repo, and no in-repo number here is a deployment answer. Slice 1 measured that the in-product inspector would have answered⚠️ This exit is available to runs stranded from here on, not retroactively. Rescuing already-stranded rows is a data question (their
0before its widening, so an in-product zero was not evidence either. Sizing still needs an operator census oversys_automation_run(status='failed') joined againstsys_approval_request— and note this verb changes nothing about that: a run stranded before this ships carries no snapshot, so it answersNO_CONSUMED_SUSPENSION.variableswere deleted with their paused row) and is not something code in this repo can answer.Boundaries honoured
⛔ The resume ordering,
forgetSuspendedRunandtraverseNextare untouched — pinned, not asserted. ⛔ No platform-level name or status for the condition. ⛔ The approvals reject door is untouched, and so is the ledger #13568 covers. ⛔ Nothing undercontent/docs/releases/. #13937 stays open and unruled; #13807 is not addressed here.Generated by Claude Code