Skip to content

Rust #325: retry children and keep their timeout options as published PHP does #869

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rmcdaniel merged 3 commits into from
Oct 29, 2026
Merged

Rust #325: retry children and keep their timeout options as published PHP does #869

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
8c2e217
Retry children and keep their timeout options as published PHP does
rmcdaniel Oct 29, 2026
f6c58c7
Start a child retry on MySQL and MariaDB too
rmcdaniel Oct 29, 2026
eb7abf8
Refuse a child deadline or backoff too far away for a timestamp
rmcdaniel Oct 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .github/workflows/server-parity.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ on:
- tests/Unit/ServerParityManyUpdatesContractTest.mjs
- tests/Unit/ServerParityWorkerErrorBodiesContractTest.mjs
- tests/Unit/ServerParityChildDescriptionsContractTest.mjs
- tests/Unit/ServerParityChildRetriesContractTest.mjs
- tests/Unit/ServerParityStartSearchAttributesContractTest.mjs
- tests/Unit/ServerParityQueryArgumentsContractTest.mjs
- tests/Unit/ServerParityQueryContractArgumentsContractTest.mjs
Expand Down Expand Up @@ -1991,7 +1992,7 @@ jobs:
bash scripts/ci/run-composer.sh --rm -v "$PWD:/source" -w /source -- \
install --working-dir=scripts/conformance/server-parity --no-interaction --no-progress
docker run --rm --user=1000:1000 --entrypoint node -v "$PWD:/source:ro" -w /source \
"$image" --test tests/Unit/ServerParityRunnerTest.mjs tests/Unit/ServerParityRunnerBlockedExcerptTest.mjs tests/Unit/ServerParityProbeNotesTest.mjs tests/Unit/ServerParitySdkDirectionLanesTest.mjs tests/Unit/ServerParityCodecDeadlineCellsTest.mjs tests/Unit/ServerParityCooperativeContractTest.mjs tests/Unit/ServerParityChildCancellationContractTest.mjs tests/Unit/ServerParityScheduleContractTest.mjs tests/Unit/ServerParityVisibilityContractTest.mjs tests/Unit/ServerParityAdmissionContractTest.mjs tests/Unit/ServerParityRoleTokenContractTest.mjs tests/Unit/ServerParityWorkerDeregistrationContractTest.mjs tests/Unit/ServerParitySearchAttributeContractTest.mjs tests/Unit/ServerParityWorkerListingContractTest.mjs tests/Unit/ServerParityWorkerReplacementContractTest.mjs tests/Unit/ServerParityPatchDeploymentContractTest.mjs tests/Unit/ServerParityWaitingHistoryContractTest.mjs tests/Unit/ServerParityTaskRetryContractTest.mjs tests/Unit/ServerParityPublishedPatchContractTest.mjs tests/Unit/ServerParityPublishedPatchTransportTest.mjs tests/Unit/ServerParityRecordFileTest.mjs tests/Unit/ServerParityFailureLocationTest.mjs tests/Unit/ServerParityFailureExcerptTest.mjs tests/Unit/ServerParityLateWorkflowCompletionContractTest.mjs tests/Unit/ServerParityStorageTestContractTest.mjs tests/Unit/ServerParitySchedulePoliciesContractTest.mjs tests/Unit/ServerParitySignalsBeforeWorkerContractTest.mjs tests/Unit/ServerParityScheduleCatalogContractTest.mjs tests/Unit/ServerParityOperatorMetricsContractTest.mjs tests/Unit/ServerParityActivityLeaseRenewalContractTest.mjs tests/Unit/ServerParityConcurrentWaitsContractTest.mjs tests/Unit/ServerParityContinueTransfersContractTest.mjs tests/Unit/ServerParityChildDescriptionsContractTest.mjs tests/Unit/ServerParityWorkerErrorBodiesContractTest.mjs tests/Unit/ServerParityLargeCommandBatchesContractTest.mjs tests/Unit/ServerParityManyUpdatesContractTest.mjs tests/Unit/ServerParityStartSearchAttributesContractTest.mjs tests/Unit/ServerParityQueryArgumentsContractTest.mjs tests/Unit/ServerParityQueryContractArgumentsContractTest.mjs tests/Unit/ServerParityParentCloseCancellationContractTest.mjs tests/Unit/ServerParityRecordLocalActivityContractTest.mjs tests/Unit/ServerParityActivityCancellationAckContractTest.mjs tests/Unit/ServerParityServiceCatalogContractTest.mjs tests/Unit/ServerParityServiceEndpointsContractTest.mjs tests/Unit/ServerParityServiceCallsContractTest.mjs tests/Unit/ServerParityWorkflowImportContractTest.mjs tests/Unit/ServerParityWaterlineImportContractTest.mjs tests/Unit/ServerParitySystemMetricsContractTest.mjs tests/Unit/ServerParityOperatorDashboardTypesContractTest.mjs tests/Unit/ServerParityPrometheusMetricsContractTest.mjs tests/Unit/ServerParityPayloadCleanupContractTest.mjs tests/Unit/ServerParityRepairPassContractTest.mjs tests/Unit/ServerParityClusterInfoContractTest.mjs tests/Unit/ServerParityNamespaceDeleteContractTest.mjs tests/Unit/ServerParityMessageStreamsContractTest.mjs tests/Unit/ServerParityCancellationScopesContractTest.mjs tests/Unit/ServerParityWorkflowPollRequestContractTest.mjs tests/Unit/ServerParityUpdateValidationContractTest.mjs tests/Unit/ServerParityFixtureSelectionTest.mjs tests/Unit/ServerParityUpdateValidationRetryTest.mjs tests/Unit/ServerParityKilledAuthorPollTest.mjs tests/Unit/ServerParityImagePullTest.mjs tests/Unit/ServerParityAdapterCacheTest.mjs tests/Unit/ServerParityWorkerSessionsContractTest.mjs tests/Unit/ServerParityWorkerSessionActivitiesContractTest.mjs tests/Unit/ServerParityNexusOperationsContractTest.mjs tests/Unit/ServerParityBridgeAdaptersContractTest.mjs tests/Unit/ServerParityWorkflowStreamWritesContractTest.mjs > parity-evidence/comparator-tests.txt
"$image" --test tests/Unit/ServerParityRunnerTest.mjs tests/Unit/ServerParityRunnerBlockedExcerptTest.mjs tests/Unit/ServerParityProbeNotesTest.mjs tests/Unit/ServerParitySdkDirectionLanesTest.mjs tests/Unit/ServerParityCodecDeadlineCellsTest.mjs tests/Unit/ServerParityCooperativeContractTest.mjs tests/Unit/ServerParityChildCancellationContractTest.mjs tests/Unit/ServerParityScheduleContractTest.mjs tests/Unit/ServerParityVisibilityContractTest.mjs tests/Unit/ServerParityAdmissionContractTest.mjs tests/Unit/ServerParityRoleTokenContractTest.mjs tests/Unit/ServerParityWorkerDeregistrationContractTest.mjs tests/Unit/ServerParitySearchAttributeContractTest.mjs tests/Unit/ServerParityWorkerListingContractTest.mjs tests/Unit/ServerParityWorkerReplacementContractTest.mjs tests/Unit/ServerParityPatchDeploymentContractTest.mjs tests/Unit/ServerParityWaitingHistoryContractTest.mjs tests/Unit/ServerParityTaskRetryContractTest.mjs tests/Unit/ServerParityPublishedPatchContractTest.mjs tests/Unit/ServerParityPublishedPatchTransportTest.mjs tests/Unit/ServerParityRecordFileTest.mjs tests/Unit/ServerParityFailureLocationTest.mjs tests/Unit/ServerParityFailureExcerptTest.mjs tests/Unit/ServerParityLateWorkflowCompletionContractTest.mjs tests/Unit/ServerParityStorageTestContractTest.mjs tests/Unit/ServerParitySchedulePoliciesContractTest.mjs tests/Unit/ServerParitySignalsBeforeWorkerContractTest.mjs tests/Unit/ServerParityScheduleCatalogContractTest.mjs tests/Unit/ServerParityOperatorMetricsContractTest.mjs tests/Unit/ServerParityActivityLeaseRenewalContractTest.mjs tests/Unit/ServerParityConcurrentWaitsContractTest.mjs tests/Unit/ServerParityContinueTransfersContractTest.mjs tests/Unit/ServerParityChildDescriptionsContractTest.mjs tests/Unit/ServerParityChildRetriesContractTest.mjs tests/Unit/ServerParityWorkerErrorBodiesContractTest.mjs tests/Unit/ServerParityLargeCommandBatchesContractTest.mjs tests/Unit/ServerParityManyUpdatesContractTest.mjs tests/Unit/ServerParityStartSearchAttributesContractTest.mjs tests/Unit/ServerParityQueryArgumentsContractTest.mjs tests/Unit/ServerParityQueryContractArgumentsContractTest.mjs tests/Unit/ServerParityParentCloseCancellationContractTest.mjs tests/Unit/ServerParityRecordLocalActivityContractTest.mjs tests/Unit/ServerParityActivityCancellationAckContractTest.mjs tests/Unit/ServerParityServiceCatalogContractTest.mjs tests/Unit/ServerParityServiceEndpointsContractTest.mjs tests/Unit/ServerParityServiceCallsContractTest.mjs tests/Unit/ServerParityWorkflowImportContractTest.mjs tests/Unit/ServerParityWaterlineImportContractTest.mjs tests/Unit/ServerParitySystemMetricsContractTest.mjs tests/Unit/ServerParityOperatorDashboardTypesContractTest.mjs tests/Unit/ServerParityPrometheusMetricsContractTest.mjs tests/Unit/ServerParityPayloadCleanupContractTest.mjs tests/Unit/ServerParityRepairPassContractTest.mjs tests/Unit/ServerParityClusterInfoContractTest.mjs tests/Unit/ServerParityNamespaceDeleteContractTest.mjs tests/Unit/ServerParityMessageStreamsContractTest.mjs tests/Unit/ServerParityCancellationScopesContractTest.mjs tests/Unit/ServerParityWorkflowPollRequestContractTest.mjs tests/Unit/ServerParityUpdateValidationContractTest.mjs tests/Unit/ServerParityFixtureSelectionTest.mjs tests/Unit/ServerParityUpdateValidationRetryTest.mjs tests/Unit/ServerParityKilledAuthorPollTest.mjs tests/Unit/ServerParityImagePullTest.mjs tests/Unit/ServerParityAdapterCacheTest.mjs tests/Unit/ServerParityWorkerSessionsContractTest.mjs tests/Unit/ServerParityWorkerSessionActivitiesContractTest.mjs tests/Unit/ServerParityNexusOperationsContractTest.mjs tests/Unit/ServerParityBridgeAdaptersContractTest.mjs tests/Unit/ServerParityWorkflowStreamWritesContractTest.mjs > parity-evidence/comparator-tests.txt
docker run --rm --user=1000:1000 --cpus=1 --memory=256m --memory-swap=256m \
--network=none --entrypoint node -v "$PWD:/source:ro" -w /source \
"$image" --test tests/Unit/ServerParityCodecDeadlineLoadTest.mjs > parity-evidence/codec-load-tests.txt
Expand Down
71 changes: 71 additions & 0 deletions docs/rust-server-port-checkpoints/2026-10-29-child-retries.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
# October 29, child retry and timeout options

- **Before.** Native refused `retry_policy`, `execution_timeout_seconds` and
`run_timeout_seconds` on `start_child_workflow` with `422
unsupported_request_field`. Its child runs never had a budget or deadline.
- **Contract** (published Server 2.11.7, Workflow 2.15.7
`ChildWorkflowRetryPolicy` and the bridge's `applyStartChildWorkflow` and
`startChildRetryIfAvailable`):
- **Recorded options.** The call, `ChildWorkflowScheduled`,
`ChildRunStarted` and the child's `WorkflowStarted` record a retry
snapshot (`snapshot_version`, `max_attempts` at least 1,
`backoff_seconds`, distinct trimmed `non_retryable_error_types`) and a
timeout snapshot. A policy with no known key records nothing. Each
timeout sets its own deadline on the child run.
- **Retry.** When a child run fails, or times out, and the call's policy
has attempts left (one per link of the call), the child instance starts
a new run under a new link. It keeps the call, the execution deadline
and the arguments, and gets a new run deadline. Its task is available
after the policy's backoff. The parent records `ChildRunStarted` with
`retry_attempt`, `retry_of_child_workflow_run_id` and
`retry_backoff_seconds`, without null fields, and keeps waiting.
- **No retry.** The failure is non-retryable when the run's failure is
marked so, or the policy names the type or class of the run's
`WorkflowFailed` event or of its failure. The parent then sees the
child fail, as it does when attempts run out.
- **Unchanged.** The parent's child wait keeps naming the first run.
- **Fixture.** `child-retries`, raw worker HTTP on protocol 1.20 in a fresh
namespace. Each case starts a parent with one child and plays the child's
attempts:
- a failure, then a completion, with a one-second backoff and both
timeouts;
- two failures with two attempts;
- a failure of a non-retryable type;
- a failure marked non-retryable;
- an empty policy;
- timeouts without a retry policy;
- two completions after a one-second run deadline.

It records the answers, the parent's child events, the child instance's
runs and their started events, the parent's description while a retry is
pending, and the parent's resumption. IDs are labels and deadlines are
recorded by presence, with how a retry's deadlines relate to the first
run's.
- **Native** (`child_retries.rs`, `children.rs`, `child_summary.rs`):
- the three options are accepted and recorded as above;
- `resolve_child` retries a failed child before the parent sees it;
- the child summary follows a retried call to its latest run and link,
so the parent stays described as waiting for the child;
- a timed-out child's `ChildRunFailed` names `closed_reason: timed_out`,
as PHP's does, while the call keeps `failed`. Native named `failed`
for any timed-out child, retried or not; the fixture's first native
recording found it.
- **Tests.** `child_retries.rs` (native), and the snapshot, backoff and
non-retryable rules in `child_retries.rs` unit tests.
- **Bounds** (Security review). A child timeout or backoff entry too far
away for a timestamp is refused at admission with `422
invalid_child_timeout` or `invalid_child_retry_policy`, and every child
deadline is computed with checked arithmetic, so an extreme value cannot
overflow. PHP's Carbon wraps such a value instead; native refuses a timer
delay or activity backoff the same way.
- **CI.** The first hosted run failed the MariaDB corpus: the retry read
`MAX(run_number)`, which MySQL returns unsigned, and repeated a
placeholder. Both are fixed. Locally, the fixture passes on MariaDB 10.11,
PostgreSQL 17 and SQLite.
- **Not covered here.**
- Cooperative cancellation delivered to a retried child: native's
delivery expects one link per call and refuses it with `409
cancellation_delivery_shape_mismatch`. Security's review confirmed this
from the source. A fix needs a PHP recording first: #873.
- Memos and search attributes inherited by child runs, retried or not
(a separate gap).
29 changes: 22 additions & 7 deletions docs/rust-server-port.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -599,10 +599,25 @@ contract fixture before its correction.
cases against published 2.11.5
([checkpoint](rust-server-port-checkpoints/2026-10-29-child-descriptions.md)).
An open activity is likewise described ahead of open children
(October 29). A child with a retry or timeout policy, or a cancellation
policy other than `abandon` under a worker that declares cooperative
cancellation, is still described without its wait, pending its own
evidence.
(October 29). A child with a cancellation policy other than `abandon`
under a worker that declares cooperative cancellation is still described
without its wait, pending its own evidence.
- **Child retry and timeout options** (October 29, closed). Published PHP
accepts `retry_policy`, `execution_timeout_seconds` and
`run_timeout_seconds` on `start_child_workflow`, records them on the call
and the child's runs, and retries a failed or timed-out child as a new run
of the same child instance while attempts remain and the failure is
retryable (`startChildRetryIfAvailable`). Native refused the three fields
with `unsupported_request_field`. It now records and retries the same
way, and describes a parent waiting on a retried child as waiting for it.
The `child-retries` fixture records seven cases against published 2.11.7
([checkpoint](rust-server-port-checkpoints/2026-10-29-child-retries.md)).
A child timeout or backoff too far away for a timestamp is refused
(`invalid_child_timeout`, `invalid_child_retry_policy`), as a timer delay
is. Remaining: native's cooperative cancellation delivery expects one
link per call, so it refuses one delivered to a retried child
(`cancellation_delivery_shape_mismatch`; #873, pending a PHP recording);
and a retried child inherits no memos or search attributes (below).
- **Trace context (Core 2.10, published in Server 2.11.1).** Core adds
nullable `observability_context` columns on runs, commands and tasks.
Native has no carrier: it accepts and ignores a top-level carrier on
Expand Down Expand Up @@ -659,9 +674,9 @@ contract fixture before its correction.
- A field for a feature native cannot run, such as `worker_session`, is
refused with `unsupported_request_field`. Fields PHP ignores are dropped
as PHP drops them.
- A timer delay, retry backoff or wait timeout too large for a timestamp
is refused on native. PHP accepts it, and wraps a timer's fire time into
the past, so the timer fires at once.
- A timer delay, retry backoff, child timeout or wait timeout too large
for a timestamp is refused on native. PHP accepts it, and wraps a
timer's fire time into the past, so the timer fires at once.
- **Search attribute limits.** Over 100 attributes or 64 KB per run, PHP
raises an uncaught `InvalidArgumentException` while applying the command.
Native refuses the completion with `422 search_attribute_limit_exceeded`
Expand Down
Loading
Loading