Repository navigation
Respect the workflow-completion aggregate limit without external storage #254
Description
The published Rust 4.15.0 worker refuses a 300 × 10 KB activity fan-out on published native Server 3.8.0 when the namespace has no external payload storage. It exits with Error::Codec("external_payload_unavailable: namespace runtime storage is not available"); the run remains pending. Each argument is within the per-payload limit and the roughly 4 MB encoded completion fits the Server's 32 MiB aggregate allowance.
This is a second Rust-specific cause of the accepted cross-repository defect #228, separate from Server #1136's early HTTP 413 response loss. Published update-outcome/null controls and ordinary lifecycle/deadline acceptance pass; no successful fan-out control is claimed yet. The installed receipt and diagnostic progression are recorded in #251.
src/runtime_uploads.rs reads only limits.max_payload_bytes into Policy.request_bytes and uses it as the whole completion's aggregate cap. A batch above the ordinary 2 MiB cap is planned for externalization even when each value is small and the native completion endpoint accepts it inline. The missing namespace store then stops the worker before delivery.
Acceptance:
- Distinguish the advertised workflow-completion aggregate limit from the ordinary request/per-payload limits. Preserve one atomic command batch, per-value checks, storage/lease authority and existing ordinary-request behavior.
- Negotiate the higher allowance explicitly; do not infer it from a Server version or assume the native multiplier for older Servers. Coordinate any missing discovery field with Server.
- A real installed worker completes 300 × 10 KB arguments and records exactly 300 completed activities without external storage on the qualified published tuple.
- An unsupported aggregate has a typed structural-limit outcome and does not become an endless retry or replay block. Keep the known Server transport defect explicit until its confirming published run.
Owner: sdk-rust. Next action: agree the discovery field with Server, add focused failing policy/route tests, implement the scoped limit, and repeat installed acceptance after publication.
Activity
rmcdaniel commentedon Nov 13, 2026
Planner contract agreed with Server:
limits.max_workflow_task_completion_bytesis an optional positive integer wire-request cap, used only for workflow-task completion and activity continuation completion. Honor a smaller explicit cap too. Absent retains ordinary-cap fallback; never infer a version or 16× multiplier. Rust's existing strict discovery parser will reject explicit malformed/null/zero/negative fields withexternal_payload_unsupportedbefore any upload/submission; coordinating the same shared vectors with PHP.When storage is unavailable and each inline value satisfies the existing per-value/threshold rules, preserve the whole atomic inline completion. If it exceeds the advertised aggregate cap, send it once and retain the Server's authoritative typed
workflow_task_completion_bytesrefusal. Do not synthesize an HTTP status or turn an aggregate limit into a demand for unavailable external storage. Ordinary requests and individual values retain their existing limits.Rust implementation owner: sdk-rust, #254. Source tests will cover both routes, unsupported/invalid producers, per-value/ordinary isolation, no partial uploads and actual typed HTTP 413 handling. Published fan-out and over-limit acceptance remain outstanding until the corrected published tuple is available.
rmcdaniel commentedon Nov 13, 2026
Implementation is in draft PR #255 at 5338f45. Owner sdk-rust: drive exact-head CI and independent review, then qualify/release against Server #1203 after its publication. Full local suite passes 1000 tests; the new real-worker feature fixture is compiled but has not run against a qualified published producer. Workflow completion is the implementation scope; raw continuation offload and lease authority remain unchanged. This issue stays open until exact installed fan-out and over-limit acceptance plus downstream delivery.
rmcdaniel commentedon Nov 13, 2026
PR #255 merged at
3e8b28299528b0d49159632309f13a1ccdec2ad5after Security's exact-head PASS and all required checks. The remote head is deleted and confirmed absent. The implementation and regression fixture are public; raw continuation offload/lease authority remain unchanged.Owner sdk-rust. Release draft #256 prepares 4.16.0 with this fix and revision-2 metrics. Exact package/lifecycle backward-compatibility qualification is running against published native 3.8, which cannot qualify the new negotiated-cap fan-out path. Publication and feature delivery remain held for Server #1203's qualified published producer and the real installed-worker within/over-cap cell. Server owner and Security have the dependency and test contract; sdk-rust owns the SDK package/installed repeat and downstream evidence. This issue stays open.
rmcdaniel commentedon Nov 13, 2026
Final open-item audit: source fix #255 is merged and release draft #256 is all-green at afc2b2f, with exact package identity plus older-producer lifecycle/deadline and metrics controls passing. Owner sdk-rust will run the real within/over-cap Worker cell against the corrected published Server tuple, then drive final package review/publication, registry repeat and downstream delivery. Server owns #1203 publication (planned 3.9.2 after 3.9.1) and has been asked for the exact index/platform/source/version tuple; its owner confirmed the ordering. Security owns the requested independent source/package verdict. No source or 3.8 image result substitutes for feature acceptance; #254 remains open.
rmcdaniel commentedon Nov 13, 2026
The release dependency is now the combined corrected native Server 3.9.1 cohort (#1202 + #1203), as confirmed by the Server owner and Security. The tentative 3.9.2 split is superseded. The durable cohort decision retains combined CI, counter-review, final notes/tree and immutable publication gates. This is an unreleased plan, not a version-based capability assumption.
SDK source
afc2b2ff8960ac2e080ffec765c66f59200cd544and candidate package89f6418ca72e1c6e13495281aab131417c5ddcee1946bfc1a22b1e63cc2b11ccremain unchanged. Security has granted exact-head release-source PASS; all current checks are green. Final tag, approval record and publication remain held until Server provides its qualified immutable index/platform/source/version tuple and the candidate passes actual installed within/over-limit Worker acceptance. The consumer must see the explicit discovery field, satisfy per-value/ordinary limits and complete a same-worker follow-up. Neither older Server evidence nor the planned version number substitutes for that gate.Owners: Server drives combined CI/counter-review/publication and sends the exact tuple; sdk-rust runs the real feature repeat and protected package/publication/registry follow-through; Security retains final package review. Existing baseline and metrics candidate PASS evidence stays scoped to published Server 3.8.0.
Release preparation PR #256 is now merged at
154fab4dcf5e503328a42c61932bcaa4b5af6523after exact-source Security PASS and green required checks; its remote head is deleted and confirmed absent. The approved release-source candidate remainsafc2b2ff, with unchanged package bytes. The real completion-limit fixture also compiles as a fresh archive-only consumer; execution remains held for the qualified corrected published Server tuple. No tag, approval record or publication exists.rmcdaniel commentedon Nov 13, 2026
Owner sdk-rust is running the real-worker within/over/follow-up cell using exact candidate source
afc2b2ff8960ac2e080ffec765c66f59200cd544/ package89f6418ca72e1c6e13495281aab131417c5ddcee1946bfc1a22b1e63cc2b11ccagainst the now-qualified published native Server 3.9.1 tuple: source7d7ad9d6c9aee2e4592d74389ba9efd36e741f44, indexsha256:946c65eb70e8b4b7c4694be8a181c9669d46c3408b80d4e44d96f2030ea3cf51, executed Linux amd64 childsha256:b0f2bf2a37e8b229282362acb26d9ed511596ebdd366ea4e53bd9f16008a356e.The complete raw index hash and child selection were independently checked. Server publication/corpus/four-cell Compose evidence has Security's final countercheck. This does not substitute for the SDK cell: the archive-only consumer must complete 300 × 10 KB inline arguments, fail a 60 × 450 KB aggregate structurally without partially scheduling activities, then complete another workflow on the same worker. The fixture checks explicit default discovery caps and unconfigured storage, not version inference.
One bounded disposable runtime and one two-job Rust consumer; all state is synthetic and disk-backed. Tag/approval/publication remain held until this result plus exact final package review. Registry acceptance is a separate post-publication repeat. No success is claimed yet.
rmcdaniel commentedon Nov 13, 2026
First real-worker candidate run on the qualified Server 3.9.1 tuple is NONPASS: runner shutdown bound, not release acceptance. All assertions for 300 × 10 KB completion, one structural WorkflowFailed with no partially scheduled activities, and the same-worker follow-up returned successfully. The test then failed at its final
tokio::time::timeout(15 seconds, running)(tests/completion_limits_server.rs:163,Elapsed). The fixture had retained the Worker's default 30-second poll timeout, inconsistent with that shorter shutdown bound.Owner sdk-rust is correcting only the public acceptance fixture to use the established short-poll setting already used by the metrics qualifier. Every business outcome, history, authority and shutdown assertion remains. The original failed log is preserved; the runtime's automatic cleanup left no owned container. This is not a PASS and does not authorize a tag or publication. Package/source
afc2b2ff/89f6418cis unchanged; a separate runner-source review and confirming repeat will follow.rmcdaniel commentedon Nov 13, 2026
Corrected-runner real-worker candidate acceptance PASS, separate from the preserved original shutdown NONPASS.
Exact tuple: SDK package
89f6418ca72e1c6e13495281aab131417c5ddcee1946bfc1a22b1e63cc2b11cc, clean sourceafc2b2ff8960ac2e080ffec765c66f59200cd544; published native Server 3.9.1 source7d7ad9d6c9aee2e4592d74389ba9efd36e741f44, indexsha256:946c65eb70e8b4b7c4694be8a181c9669d46c3408b80d4e44d96f2030ea3cf51, Linux amd64 childsha256:b0f2bf2a37e8b229282362acb26d9ed511596ebdd366ea4e53bd9f16008a356e. Runner #257 isb3480de83b30764b6d4237f5de243a49d9a22f61, test-file SHA-256907b5410cffa22110fb852ab9b9f37780d21fa0665328eebc92c3c71364c798b.The fresh archive-only consumer passed all assertions: explicit advertised ordinary/aggregate limits and no payload store; 300 × 10 KB inline arguments and exactly 300 ActivityCompleted events; a 60 × 450 KB batch yielding one non-retryable structural-limit WorkflowFailed, zero ActivityScheduled and zero ActivityCompleted events; the same worker's subsequent successful workflow; bounded clean shutdown. Test result: 1 passed, 0 failed, 64.75 seconds. Runtime cleanup passed and no owned container remains. Package/runtime bytes are unchanged from the first attempt; only the reviewed runner poll timeout differs.
This is pre-publication candidate evidence. Full package lifecycle/deadline and metrics checks are being bound to this same corrected Server tuple for final package review. Required runner CI, final independent package approval, protected publication, registry identity readback and the exact published-crate feature repeat still gate delivery. #254/shared #228 are not closed by this candidate run.
rmcdaniel commentedon Nov 13, 2026
Published 4.16.0 installed acceptance: PASS
Exact published tuple: Rust 4.16.0, registry/archive SHA-256
89f6418ca72e1c6e13495281aab131417c5ddcee1946bfc1a22b1e63cc2b11cc, clean source/tagafc2b2ff8960ac2e080ffec765c66f59200cd544; native Server 3.9.1, source7d7ad9d6c9aee2e4592d74389ba9efd36e741f44, indexsha256:946c65eb70e8b4b7c4694be8a181c9669d46c3408b80d4e44d96f2030ea3cf51, executed Linux amd64 childsha256:b0f2bf2a37e8b229282362acb26d9ed511596ebdd366ea4e53bd9f16008a356e. Public runner fixture is reviewed commitb3480de83b30764b6d4237f5de243a49d9a22f61(#257); package/source bytes remain the approved ones.Fresh registry consumer: exact lock source/checksum verified, no patch, vendor or local Cargo config. Actual tests against the same disposable published Server:
Separate registry-only lifecycle/deadline acceptance also passes. Every run cleaned its owned containers/networks; no leftovers. The prior candidate shutdown NONPASS is preserved and is not substituted for this result.
Publication 43432093783, final Release, and API docs 43432730516 all pass. Fresh API build identity is 4.16.0 / checksum above / source above / tooling
ab8cae5eeb9c2aa701a997a2593e0ecd933a57a4.Scope: the larger allowance requires explicit
limits.max_workflow_task_completion_bytes; older producers retain the conservative ordinary cap. This tuple qualifies the Rust completion/metrics behavior, not other SDKs, every backend, #216 caller-key retries or native timeout-cleanup. Root docs adoption is being driven by its owner. Sample App keeps its already-frozen 4.15 ordinary cohort, which does not claim this large-fanout slice; no redundant tuple run is required for that unchanged scope.rmcdaniel commentedon Nov 13, 2026
Published feature acceptance and the independent post-publication identity gate are now complete. Security independently verified the fresh archive, sparse checksum/not-yanked status, source/tag, strict approval record and live API identity: countercheck. The published Worker/metrics receipt remains the actual feature proof. Final Release notes now name that qualified tuple and link the receipt instead of the superseded prospective publication requirement; rendered note readback matches.
Remaining user-visible action: root docs PR #535 selects Rust 4.16 and documents the planner and seventh histogram. I counter-reviewed its bounded Rust/manifest wording at exact
ad73072319b45096e7b9c04d5d26fda00412b3b5: semantic PASS. All current hosted checks pass; the docs owner is completing its independent/all-locale review and protected deployment.Next action and owners: durable-workflow-github-io merges/deploys #535 and sends the live manifest/routes receipt; sdk-rust independently reads back that live selection and affected Rust paragraphs, records it here and closes this issue. The docs owner has been asked directly. Sample App intentionally retains its already-frozen ordinary 4.15 cohort; it does not claim this large-fanout feature, so no redundant tuple update blocks this scope. Caller-key #216 and native timeout cleanup remain separate.
rmcdaniel commentedon Nov 13, 2026
Delivered in published Rust 4.16.0 with qualified native Server 3.9.1. Registry-installed Worker/metrics acceptance, independent registry/tag/approval/API identity PASS, and the final 4.16 Release bind the exact package/source/index/platform identities.
Remaining downstream adoption is now live. Root docs #535 merged as
ff481879798f7d4fec0f5b3eeb85690e9f67584d; protected Pages run 43440193660 succeeds. The docs owner verified all 40 affected pages across eight locales and unchanged installers: live receipt.I independently fetched the public surfaces with cache revalidation at 2026-11-13 14:25 UTC:
89bd73307c750b3f82270104493345e7e133f937e1c3325084103af29d8586cb.Sample App intentionally retains its already-frozen 4.15 ordinary cohort, which does not claim this large-fanout slice; no redundant tuple change is required for that scope. This closes the Rust issue's implementation, publication, installed acceptance and relevant live documentation follow-through. Shared #228's other SDK/backend cells and #216 remain owned separately.