Skip to content

Native: honour an activity's cancellation policy on whole-run delivery - #969

Merged
rmcdaniel merged 8 commits into
mainfrom
rust-958-activity-cancel-wait
Nov 3, 2026
Merged

rmcdaniel merged 8 commits into
mainfrom
rust-958-activity-cancel-wait

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Nov 2, 2026 •

Copy link
Copy Markdown
Member

Review tier B: new write-path behavior in cooperative delivery, the stop acknowledgement and activity scheduling; no tier-A area. The design was posted on #958 (6589297313).

Fixes #958. abandon is split to #971 (Security's review, 6590926349).

Change

Following PHP Core 2.15 (WorkflowCommandNormalizer, ActivityCancellationWait, ActivityCancellationAcknowledgement; Core 637ea0b5):

  • Accepting the policy. schedule_activity accepts cancellation_policy. Native refused it before, so every activity was try_cancel.

    • Any explicit value needs protocol 1.20 and a cooperative claim, otherwise 409 activity_cancellation_policy_not_supported with PHP's guard body.
    • An explicit abandon is refused with the same 409 plus unavailable: ["installed_runtime_activity_policy"], until abandoned activities can outlive their run (Native: abandoned remote activities must outlive their run (cancellation_policy abandon) #971).
    • The policy is stored in activity_options, so ActivityScheduled.activity.cancellation_policy names it. An abandoned activity also names schedule_to_close_deadline_at.
  • Delivery. Whole-run delivery to an activity, alone or as a parallel or selection member, follows the policy:

    • try_cancel: unchanged.
    • wait_cancellation_completed: cancels the live attempt. Until every targeted execution has resolved, it holds: the task gets a cancellation_activity_wait marker, the run waits, the claim is released, and the answer is delivered:false with reason cancellation_waiting_for_activity.
    • abandon can't be scheduled yet (Native: abandoned remote activities must outlive their run (cancellation_policy abandon) #971). A defensive delivery branch still leaves such an activity uncancelled.
  • Resume. The stop acknowledgement creates one ready workflow task (resume_source_kind: activity_cancellation_acknowledged) when:

    • every held execution has resolved;
    • the run still waits on this request, before its cleanup deadline;
    • no delivery and no open workflow task exist.

    Redelivery then delivers. A duplicate acknowledgement resumes nothing. Without an acknowledgement, the existing cleanup-deadline repair closes the run.

  • The decision is recorded in docs/rust-server-port.md.

Checked

  • a_waiting_activity_holds_the_run_delivery_until_its_stop_is_acknowledged: hold, no task before the receipt, one resumed task, a duplicate receipt, redelivery.
  • an_explicit_abandon_or_a_pre_1_20_policy_is_refused_at_scheduling: abandon gets 409 installed_runtime_activity_policy; a 1.19 policy gets 409 request_protocol.
  • The existing retried-activity acknowledgement test is unchanged and passes.
  • The full SQLite suite passes (673 execution, 397 lib). CI runs PostgreSQL and MySQL.

Not covered by a test yet

A parallel group waiting on several acknowledgements; a late acknowledgement after the cleanup deadline. The code handles both.

🤖 Generated with Claude Code

…958)

Delivery side and resume, following PHP Core's ActivityCancellationWait:
- the activity branch and parallel/selection members read the activity's
  cancellation_policy;
- wait_cancellation_completed holds the delivery (cancellation_activity_wait
  marker on the task, run waiting, claim released) until every listed
  execution has resolved;
- the stop acknowledgement resumes the run with one ready workflow task;
- abandon leaves the activity running (needs a finite schedule-to-close
  deadline).

Not done yet: native's schedule_activity does not accept cancellation_policy
at all (store.rs field list), so the new tests fail at scheduling. Next:
accept the explicit policy with Core's protocol-1.20 gate and abandon
precondition.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…#958)

schedule_activity now accepts cancellation_policy, with Core 2.15's rules:
- any explicit value needs protocol 1.20 and a cooperative claim, else 409
  activity_cancellation_policy_not_supported (PHP's guard body);
- abandon needs an integer schedule_to_close_timeout of at least 1, else 422
  validation_failed under commands.N.schedule_to_close_timeout.
The policy is stored in activity_options, so ActivityScheduled's activity
snapshot names it; an abandoned activity's snapshot also names its
schedule_to_close_deadline_at.

With the delivery and resume work, the wait and abandon scenarios pass, and
an unbounded abandon or a 1.19 policy are refused. The decision is in the
port log.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rmcdaniel
rmcdaniel marked this pull request as ready for review November 2, 2026 22:51
@rmcdaniel

Copy link
Copy Markdown
Member Author

Security bounded review at exact head 2ed4b7609dbc5f4e7c962c0e395e5caab8f4e3e0 found one release-blocking lifecycle gap in the newly admitted abandon policy.

The scheduling guard and durable snapshot match the locked Core source, but Native does not yet preserve an abandoned remote activity after the parent run reaches its canonical cooperative-cancelled terminal state:

  • cancel_activity_member leaves a pending/running abandon execution and task unchanged, which is the right first step.
  • Both Native activity candidate queries still join workflow_runs and require the parent status to be pending, running, or waiting (TASK_CANDIDATES_SQL in store.rs, and the cross-kind poll query). Therefore an abandon activity that is still ready when the workflow closes becomes invisible to every activity poller.
  • For an already leased activity, complete_activity proceeds through active_run; a canonically cancelled parent is rejected as run_closed. The analogous failure/heartbeat/lease paths need the same audit. Thus the activity can be left leased at delivery, as the new test proves, but it cannot durably finish after the workflow observes cancellation and closes.

That differs from the exact locked Workflow 2.15.12 authority (7c80ffcc…). Its DefaultActivityTaskBridge::poll does not exclude activity tasks by terminal parent status, and ActivityAbandonment::allows explicitly retains authority after a canonical cooperative RunStatus::Cancelled close while requiring the matching ActivityScheduled policy/deadline and terminal activity history. The finite schedule-to-close lifetime exists so that independent post-run work can finish or time out; stranding the task until cleanup is not Abandon behavior.

The current regression masks both cases: cancel_with_policy polls/leases the activity before requesting cancellation, and an_abandoned_activity_keeps_running... stops immediately after delivery without closing the parent or submitting an activity outcome.

Please add the narrow, fail-closed post-run authority rather than generally admitting activity work under terminal runs. It must require the exact canonical abandon snapshot/deadline and the matching cooperative cancelled-close identity. Cover at least:

  1. an unclaimed ready abandon activity remains pollable after the parent canonically cancels;
  2. a previously leased abandon activity can complete/fail after that close and records its terminal history;
  3. ordinary activities and non-canonical completed/failed/terminated/corrupt-history cases remain refused;
  4. expiry still closes the activity at its original schedule-to-close deadline.

This finding holds the exact head. The PR is also behind current main and its Sonar gate is failing, but those are separate from this product blocker.

rmcdaniel and others added 3 commits November 2, 2026 17:17
… their run

Security's review of 2ed4b76: native ties activity work to an active parent
(both activity candidate queries require a pending, running or waiting run,
and completion goes through active_run). An abandoned activity would be
stranded once the run closes, which isn't Abandon. Until #971 gives
abandoned activities their post-run authority, native refuses an explicit
abandon at scheduling with PHP's runtime refusal: 409
activity_cancellation_policy_not_supported with unavailable
["installed_runtime_activity_policy"]. try_cancel and
wait_cancellation_completed are unchanged.

The now-unreachable abandon timeout check is removed. The test covers the
abandon refusal and a pre-1.20 policy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…etup

Sonar flagged 5.1% duplicated new lines (limit 3%): the activity policy
guard repeated the child guard's protocol check and 409 body, and the tests
repeated the cooperative worker registration. cancellation_policy_guard now
serves both. Child keeps its protocol parsing; activity requires a strict
1.N version and adds installed_runtime_activity_policy for abandon (#971).
register_cooperative replaces the repeated registration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rmcdaniel

Copy link
Copy Markdown
Member Author

Security re-review of exact b7154f6: the narrowed implementation is now fail-closed for explicit abandon, but the authoritative record still claims the removed behavior. The new November 2 decision in docs/rust-server-port.md says abandon only needs a finite schedule_to_close_timeout and delivers without cancelling, while store.rs now returns 409 activity_cancellation_policy_not_supported with unavailable [installed_runtime_activity_policy]. The completion-rule descriptor also still says remote abandon requires a finite schedule_to_close_timeout. Those statements would advertise a lifecycle this release deliberately does not support and conflict with #971. Please update both durable descriptions to say try_cancel and wait_cancellation_completed are admitted under the 1.20/cooperative gate, while abandon is refused with the runtime-gap reason until #971 supplies post-run authority. The implementation narrowing itself is consistent with the original finding; this exact head remains held for the record correction and terminal hosted checks.

…fuses abandon

Security's re-review of b7154f6: the port-log decision and the
cancellation_policy field description still described abandon as supported.
Both now say native admits try_cancel and wait_cancellation_completed under
the protocol-1.20 cooperative gate and refuses abandon with
installed_runtime_activity_policy until #971.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rmcdaniel

Copy link
Copy Markdown
Member Author

Security follow-up on exact 8ff8270: the two-file correction matches the narrowed implementation. The durable decision now states that only try_cancel and wait_cancellation_completed are admitted under the protocol-1.20/cooperative gate, and that abandon is refused with unavailable [installed_runtime_activity_policy] until #971 supplies post-run authority. The completion field guidance says the same. diff --check is clean. This resolves the record mismatch and the original abandon lifecycle finding is fail-closed at this source. Final merge clearance remains pending terminal hosted checks and the ordinary carry onto post-#957 main; no need to hold 3.0.2 for this optional item.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rmcdaniel

Copy link
Copy Markdown
Member Author

Security post-#957 carry review at exact fef3247d60d39ea06f81658463c7b1b97c09fa5b: its parents are the previously reviewed 8ff827065ae88e71eea969255f77ae8ab0f9c394 and exact merged main 9524e80b2f88e6282aacbcd51560a080302d2d33; the merge has no combined-diff conflict resolution. The reviewed PR delta is content-equivalent across the carry (stable patch ID 899a532629d3a96a1667c5d5648c3899c7103ab9 before and after). Source carry is clear; final merge clearance still waits terminal hosted checks on this exact head. This PR remains optional for 3.0.2 and must not hold the customer-blocking patch.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security hosted-result hold on exact head fef3247d60d39ea06f81658463c7b1b97c09fa5b: this is a deterministic PostgreSQL test defect in the new cancellation-policy scenario, not the earlier workflow-stream flake.

Both PostgreSQL native-test matrices in run 41548607472 fail the same new test, activity_cancellation_ack::a_waiting_activity_holds_the_run_delivery_until_its_stop_is_acknowledged, at tests/execution/task_retry.rs:39 with PostgreSQL 42883: operator does not exist: json ~~ unknown. The new assertion at activity_cancellation_ack.rs:260 sends this backend-neutral SQL through task_retry::rows:

... AND payload LIKE '%activity_cancellation_acknowledged%'

workflow_tasks.payload is PostgreSQL json, so LIKE (~~) is undefined without a type-aware/portable assertion. The two failures are independent PostgreSQL matrices and otherwise report 677 passed / 1 failed / 1 ignored. This does not implicate #965/#968 and #969 was already optional for 3.0.2.

Please replace that assertion with a cross-engine-safe check (or prove the duplicate receipt by polling/counting the actual resumed task without substring matching a JSON serialization), counterfactual it on PostgreSQL, and rerun. #969 is not Security-final-cleared at this head. Do not hold the query/describe patch for it.

The wait-cancellation test counted resumed tasks with `payload LIKE ...`.
On PostgreSQL workflow_tasks.payload is json, which has no LIKE operator
(42883), so both PostgreSQL native matrices failed it. The test now reads
the payloads as text on each engine (payload, payload::text,
CAST(payload AS CHAR)) and counts them in Rust. Test-only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rmcdaniel

Copy link
Copy Markdown
Member Author

Security re-review: the PostgreSQL test hold is closed at exact head 0c44ee4c3b2f7d1e808285327c9b8216cf4fa739.

The corrective commit is a single test-only child of the held fef3247d… and changes only rust/tests/execution/activity_cancellation_ack.rs. It no longer applies LIKE to PostgreSQL json: the existing cross-engine text_rows helper selects the target run's task IDs/payloads as native text, payload::text, or CAST(payload AS CHAR), and Rust counts the exact cancellation-acknowledgement marker. The assertion remains == 1, so it still proves a duplicate receipt did not enqueue a second resumed task. No runtime or public surface changed.

This is exact-head source clearance, supported by the reported SQLite/PostgreSQL/MySQL targeted runs. The fresh hosted matrix has only just restarted, so #969 is not yet Security-final-cleared for merge; terminal green on this exact head remains required, and it must not hold #965/#968/#974 or the 3.0.2 customer-blocker path.

@sonarqubecloud

sonarqubecloud Bot commented Nov 3, 2026

Copy link
Copy Markdown

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security final carry for exact head 0c44ee4c3b2f7d1e808285327c9b8216cf4fa739: clear for merge and inclusion in 3.0.2 if the queue window remains open.

The narrowed runtime design remains exact: protocol-1.20 plus cooperative capability admits only try_cancel and wait_cancellation_completed; explicit abandon fails closed with activity_cancellation_policy_not_supported / unavailable installed_runtime_activity_policy until #971 supplies its full lifecycle. The policy is stored/reported for admitted modes, wait semantics retain one released-claim resume turn, and the final PostgreSQL correction is test-only through the existing per-engine text-row helper.

Hosted qualification is terminal on this head: 58 contexts, 57 success and the expected prepared-local cancellation context skipped, with no failure/cancellation. This closes Security's source and hosted-check holds. If a later carry merges new main, re-establish the exact head before queueing.

@rmcdaniel
rmcdaniel added this pull request to the merge queue Nov 3, 2026
Merged via the queue into main with commit 0e6fc80 Nov 3, 2026
58 checks passed
@rmcdaniel

Copy link
Copy Markdown
Member Author

Security generated-merge and post-merge carry for exact commit 0e6fc80a34941098696c53f42289874cf9f98d15: accepted as the 3.0.2 source tip.

  • The GitHub-verified merge has exactly cleared main 6760837e677275d1b65e609332a522069c9fa47c (Native: offline backup-hold release and read-only inspect for restores (#973) #974) and Security-cleared PR head 0c44ee4c3b2f7d1e808285327c9b8216cf4fa739 as parents.
  • Relative to that main parent, it changes only the six reviewed Native: honour an activity's cancellation policy on whole-run delivery #969 paths. Every result blob is byte-identical to the cleared head; there is no conflict resolution or source drift.
  • All three merge-group workflows are terminal success on this exact commit: Shared Server fixtures 41562369513, PHPUnit Feature 41562369704, and Public Boundary 41562369665.
  • The terminal merge-group inventory was 52 contexts: 51 successful, the expected prepared-local cancellation context skipped, no failure/cancellation.
  • GitHub merged the same generated commit unchanged at 2026-11-03T02:14:49Z; mergeCommit.oid and protected main both name 0e6fc80a….

Fresh post-push workflows subsequently registered on the same commit; they are additional main evidence, not a changed tree. The source/queue carry is closed. Release dispatch still requires Security's immediate pre-tag dockerhub-native policy readback, the immutable tag/source binding, publisher qualification, and independent public artifact verification.

rmcdaniel added a commit that referenced this pull request Nov 3, 2026
The update counterfactual counted update tasks with payload LIKE, which
PostgreSQL rejects on its json column (the same portability failure
#969 fixed). The two polled tasks already prove the update task remains,
so the count is replaced by the run's workflow task total (claim, update
task, turn). The child tests pass on PostgreSQL 17 (65) and SQLite.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Native ignores activity wait_cancellation_completed on whole-run cooperative cancellation

1 participant