Skip to content

Native: return the full not-recorded receipt for refused workflow-task reports (#994) - #999

Merged
rmcdaniel merged 3 commits into
mainfrom
rust-994-not-recorded-receipt
Nov 3, 2026
Merged

rmcdaniel merged 3 commits into
mainfrom
rust-994-not-recorded-receipt

Conversation

@rmcdaniel

Copy link
Copy Markdown
Member

Fixes #994

This follows Security's acceptance contract in issue comment 6613395001.

What changes

When a workflow-task report is refused task_not_leased or run_closed, nothing was written for it. The 409 now carries the full not-recorded receipt:

  • recorded: false
  • task_id: the task row's ID
  • workflow_task_attempt: the request's attempt, normalized
  • run_id: the task row's run

task_status is filled in when the refusal didn't already carry it. Every existing field stays. The receipt carries no payload or command data. The receipt identity is authoritative: it overwrites any conflicting task_id, attempt, run_id or recorded the refusal carried.

This is a native addition, not parity with PHP. Published PHP 2.11.8's lease guard (guardWorkflowTaskOwnership) sends no recorded or run_id. Only PHP's engine-outcome race path (WorkerController.php 2155–2164) carries them.

Covered surfaces

All in rust/src/runtime/store.rs, through Store::not_recorded → workflow_task_guard::not_recorded / not_recorded_answer:

Surface Refusals
Store::fail_workflow_task: workflow_task_guard error task_not_leased, run_closed (failure reports and waiting acknowledgements)
Store::fail_workflow_task: active_run error bare run_closed, now in the worker envelope
Store::complete_or_checkpoint: workflow_task_guard error task_not_leased, run_closed (completion and the shared checkpoint wrapper; includes the waiting/ack duplicate, which falls through duplicate_receipt to the guard)
Store::complete_or_checkpoint: active_run error bare run_closed
Store::complete_or_checkpoint: checkpoint_prefix answer or error Core's refused run_closed checkpoint, for a run that closed after the guard (scope_checkpoint.rs)

run_closed covers every terminal status the guard recognizes: completed, failed, timed out (status failed, closed reason timed_out), cancelled and terminated.

Unchanged (still fatal to workers)

  • workflow_task_attempt_mismatch
  • lease_owner_mismatch
  • lease_expired
  • stale_worker_registration
  • conflicting_completion
  • cancellation_deadline_expired
  • invalid commands and every other refusal

Workflow-task heartbeat and history-read refusal bodies are untouched.

Tests

rust/tests/execution/not_recorded_receipts.rs pins the public HTTP body exactly: the worker envelope, server_capabilities present, run_closed_at format-checked. Each case also proves nothing was recorded: every table's row count, the run history and the dw_task_completions receipts are unchanged.

  • a_report_on_every_closed_run_status_is_run_closed_with_the_full_receipt: completed, failed, timed_out, cancelled and terminated × complete, fail and checkpoint.
  • a_report_for_a_finished_task_on_an_open_run_is_task_not_leased_with_the_full_receipt: a completed task and the waiting/ack duplicate × complete, fail, waiting ack and checkpoint.
  • a_report_on_a_closed_run_whose_task_is_still_leased_carries_the_full_receipt: the bare active_run run_closed, on complete and fail.
  • a_checkpoint_whose_run_closed_after_the_guard_carries_the_full_receipt: a cooperative claim still leased, then its run closed (completed, failed, cancelled, terminated), through Core's checkpoint.
  • fatal_report_refusals_gain_no_receipt: attempt mismatch, owner mismatch and expired lease on all three routes keep their exact published bodies.

There are also new unit tests in workflow_task_guard.rs: fatal reasons pass through unchanged, conflicting identity is overwritten, and other checkpoint answers are unchanged. late_workflow_claims.rs now expects the receipt on the post-close fail and complete refusals only. Its heartbeat and attempt-mismatch expectations are unchanged.

Results

Run on lab3 in rust:1.99.0-slim-bookworm (--cpus 3 -m 5g, CARGO_BUILD_JOBS=2):

  • cargo test --locked --lib: 413 passed.
  • cargo test --locked --test execution: 694 of 696 passed. The 2 failures were load-timing (namespaces::a_named_namespace_child_runs…, workflow_failures::workflow_failure_attributes_only…). Both pass on rerun in isolation, twice. They don't involve these refusals. An earlier full run had 4 different intermittent failures; all of them passed on rerun.

Counterfactuals

  • origin/main sources with these tests: 4 receipt tests and late_workflow_claims fail on the missing recorded/run_id. On main the bare run_closed also lacks the worker envelope. fatal_report_refusals_gain_no_receipt passes on main, as intended.
  • Without the checkpoint_prefix mapping: the checkpoint test fails. The body has no recorded, run_id or attempt.
  • With entry().or_insert identity instead of overwrite: the authoritative-identity unit test fails.

Not in this PR

The cross-SDK conformance fixture and the implementation-aware late-workflow-completion contract are the Server agent's commit on this branch. Until that lands, the shared parity contract still expects 2.11.8's bodies for the three affected late reports.

There is no CHANGELOG in this repo.

🤖 Generated with Claude Code

#994)

A workflow-task completion, failure, waiting acknowledgement or
cancellation-scope checkpoint refused task_not_leased or run_closed has
written nothing. Its 409 now carries recorded:false, the task row's exact
task_id and run_id, and the request's normalized workflow_task_attempt,
beside the refusal's existing fields and task_status. This follows
Security's contract in issue comment 6613395001.

The guard and open-run refusals in fail_workflow_task and
complete_or_checkpoint are mapped, as is Core's refused checkpoint for a
run that closed after the guard. The identity fields are authoritative
and overwrite anything the refusal carried. Attempt and owner mismatches,
expired leases, stale registrations, heartbeats and history reads are
unchanged and stay fatal to workers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ive only (#994)

Published PHP 2.11.8's ownership guard refuses a late completion or
failure, and a closed-run report, without recorded or run_id. It reports
both only from its rarer post-guard engine outcome, which a deterministic
late request never reaches. Native now adds the not-recorded receipt to
those refusals (server#994).

The probe records the implementation from cluster info (a failed read
throws). The contract requires PHP's published bodies exactly when it is
absent, and for native the same bodies plus exactly recorded false and the
task's run_id. Renewal and attempt-mismatch refusals are unchanged for
both. Any other implementation is refused.

Unit controls: PHP with native fields, native without them (3.0.4's
bodies), a wrong run, a claimed record, receipt fields on renewal or
attempt mismatch, and unknown or unnamed implementations all fail.
Against main's contract the native case and the 3.0.4 control fail;
without the implementation check, both unknown-implementation controls
fail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rmcdaniel

Copy link
Copy Markdown
Member Author

The conformance half is added as b4654d1a33a7adb1a2288e8cc3d5a6af0a48c7a6. It's a cherry-pick of 11c1bd12, which Security cleared in chat 7333, and its git patch-id --stable matches the original (0eae48f3).

  • Probe: records implementation from cluster info; a failed read throws.
  • Contract:
    • Published PHP (implementation null) must match its published bodies exactly.
    • Native (rust) must match the same bodies plus exactly recorded: false and run_id equal to the authoritative run, on the completion, failure and closed-run refusals.
    • Renewal and attempt-mismatch bodies are unchanged for both.
    • Any other implementation is refused.
  • Unit test: tests/Unit/ServerParityLateWorkflowCompletionContractTest.mjs passes 34/34 on the combined tree, with 14 new negative controls. The counterfactuals against main's contract are in 11c1bd1's commit message.

The native corpus's late-workflow-completion case failed on 84e9374 alone, as expected. It should pass in this run.

Merge-only refresh onto main 42f6e27 (#995), at Security's request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rmcdaniel

Copy link
Copy Markdown
Member Author

Security exact-head refresh readback for a4e4a1fe134278d85ca7d3e6184f8d10940690bc:

The merge-only refresh introduces no overlap or source drift. Source and conformance remain cleared at this exact head. Final merge clearance is held only for the full hosted matrix, including native parity/corpus, to finish on a4e4a1fe with no failure or pending job.

@sonarqubecloud

sonarqubecloud Bot commented Nov 3, 2026

Copy link
Copy Markdown

@rmcdaniel

Copy link
Copy Markdown
Member Author

Hosted checks are terminal at exact head a4e4a1fe134278d85ca7d3e6184f8d10940690bc. That's the merge-only refresh onto main 42f6e278 (#995), which Security read back as source-clean in comment 6614936729.

  • Totals: 57 success, 1 expected skip, 0 failed or pending. I read these from the commit's own check runs. GitHub reports CLEAN.

  • Corpus and native parity, all success:

    • execution-and-codec (corpus and native-tests) on SQLite;
    • mysql-execution corpus and native-tests on mysql-8.0, mariadb-10.11 and the existing-php-mysql-matrix;
    • postgres-storage corpus and native-tests on postgres-17 and the existing-php-matrix;
    • codec-native-build and native-debug-build;
    • Shared Server fixtures result.

    The SQLite corpus includes late-workflow-completion under its new implementation-aware contract, with native carrying the receipt and published PHP unchanged.

  • Other checks: the PHPUnit feature suite passed.

Requesting Security's final clearance. Server will mark this ready and queue exactly this SHA.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security final exact-head clearance: a4e4a1fe134278d85ca7d3e6184f8d10940690bc is accepted to merge.

Independent readback confirms the source-cleared head is unchanged, CLEAN/MERGEABLE, with 57 terminal successes, one expected conditional skip, and zero failed or pending checks. The terminal matrix includes the native corpus and native tests on SQLite, MySQL 8, the existing PHP MySQL matrix, MariaDB 10.11, PostgreSQL 17 and the existing PHP PostgreSQL matrix; codec/native builds, feature suite, corrected markers, SDK directions and the aggregate Shared Server fixtures result all pass.

This exact head closes the #994 receipt contract: native task_not_leased and run_closed refusals that record nothing carry the authoritative recorded:false, task id, attempt and run id; fatal attempt/owner/lease/stale refusals, renewal and heartbeat bodies remain unchanged; the implementation-aware conformance fixture preserves published PHP bodies and fails closed on unknown implementations.

Merge only this SHA and return the merge commit, parents and tree readback. Server 3.0.5 and sdk-rust #170 remain downstream gates until the merged artifact is qualified.

@rmcdaniel
rmcdaniel marked this pull request as ready for review November 3, 2026 21:08
@rmcdaniel
rmcdaniel added this pull request to the merge queue Nov 3, 2026
Merged via the queue into main with commit f1cd956 Nov 3, 2026
58 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Native: return the full not-recorded receipt for late workflow-task completions

1 participant