Skip to content

Allow redrive through validated activity retry histories - #1131

Merged
rmcdaniel merged 4 commits into
mainfrom
fix/1063-redrive-retry-prefix
Nov 13, 2026
Merged

rmcdaniel merged 4 commits into
mainfrom
fix/1063-redrive-retry-prefix

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Failed runs with ordinary retried activities are refused by the redrive planner, which treats ActivityRetryScheduled as unsupported history. This extracts the generic redrive prerequisite from Saga review B1 into a main-based change: validate the recorded execution/attempt/task retry chain, reuse completed prefix outcomes without another callback, and resume the failed step. A same-run accepted active start/adopt receipt is ignored only after its command identity is verified.

Scope is the redrive planner and its ordinary-workflow regression coverage. The Saga ledger/API remains in #1110; existing service-worker definition-fingerprint requirements remain enforced. No native implementation or installed parity is claimed. The planner changes and four replay fixtures are derived from reviewed source 1d2e41203005b8ac086e436c019d17c00a079454; maintainer follow-up is tracked under #1063 and B1 review.

Validation at cebeab3b: changed-file ECS and PHPStan pass; planner unit controls pass (19 tests/77 assertions) while current main produces five expected failures with the same tests; selected replay corpus passes (17 tests/437 assertions). Added ordinary queue-job and service-fingerprint feature controls are not yet qualified. Full ECS/static/unit/coverage/feature cycle, corpus counterfactuals, hosted checks, and exact-head Security review remain merge gates. Draft until those gates are complete.

This addresses B1 only. The #1063 acceptance work remains open.

@rmcdaniel

rmcdaniel commented Nov 13, 2026 •

Copy link
Copy Markdown
Member Author

Current merge disposition: HOLD under Security's retry-grammar finding. The earlier General source countercheck below did not cover heartbeat identity/ordering and duplicate/mid-chain schedules. Security reproduced their acceptance; that finding supersedes the earlier positive countercheck. Workflow is correcting the finite-state grammar; General is adding focused predecessor-failing controls in a separate test file.

Successor c0f18b322e6f2b00c2559a9027474f7d296c8bd3 preserves the previously reviewed planner/helper bytes. Its only changes select the supported asynchronous test queue and use the supported heartbeat fixture field. I independently reran both ordinary retry feature cases on this head: 2 tests / 26 assertions pass, PHP 8.4.27, PHPUnit 13.4.6, owned SQLite, read-only source/vendor and no network. PHPUnit reports one deprecation. Those positive cases do not close the subsequent grammar finding.

The underlying review of cebeab3b15505a314f8dc8d522e2f0f45509a85a found the B1 redrive behavior isolated from the compensation ledger and its migrations, with no additional core blocker in this split.

The retry helper admits a linear started/retry/started/terminal chain for one activity execution, with increasing attempt numbers, unique attempt/task identities and exact retry predecessor/successor bindings. Completed prefix results are reused; only the failed boundary is rerun. The ordinary-workflow feature additions exercise both a retried failed boundary and a retried completed prefix, preserve the source history/deadline and repeat the same redrive request without another execution. Service-worker definition/fingerprint checks remain required and now cover a retried prefix.

The start/adopt exception is limited to an accepted returned_existing_active receipt bound to the same run, instance, workflow class/type and command. Rejected starts retain their refusal. The new unit cases cover completed retries and intervening heartbeats; the changelog and four replay fixtures address the original B1 review requirements.

I independently confirmed the current fixture problem already identified by Workflow: both new ordinary-retry feature cases select this class's sync default, so they stop at queue_sync_unsupported with zero product assertions. That run is nonpassing. Correct the fixture to the asynchronous queue configuration used by its manually delivered tasks, then retain fresh exact-head feature/full-CI results. No product guard should be relaxed for the test.

This source countercheck does not infer a passing feature run, full quality cycle, Security verdict or completion of B2–B6. Land this prerequisite through the normal protected path once its corrected head qualifies, then refresh the Saga implementation on that accepted parent.

@rmcdaniel

Copy link
Copy Markdown
Member Author

At c0f18b322e6f2b00c2559a9027474f7d296c8bd3, the two ordinary queue-job retry/redrive cases and two service fingerprint cases pass on MySQL 8.4.8: 4 tests / 62 assertions. The ordinary controls verify callback cardinality, failed-step resumption, immutable source history, unchanged execution deadline and idempotent repeated redrive. The service controls preserve missing/unverified/changed definition refusals for both ordinary and retried completed prefixes. The first fixture attempts exposed the class's forced sync default and an unsupported fixture heartbeat key; those test-only mistakes are corrected in this head.

Planner tests pass (19/77). With current main's planner substituted, the same tests fail five cases, and the four new replay fixtures fail while the candidate's selected corpus passes (17/437). Full PHPStan now passes with two bounded workers and explicit memory; the initial default-memory attempt ran out of its configured 128 MiB and is not passing evidence. Full ECS/unit/coverage/feature and current-head hosted qualification are still in progress. General owns independent counter-review; Workflow owns all remaining merge gates and exact Security review. No Saga ledger code or native implementation is enabled here.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security exact-source review of Workflow #1131 at c0f18b322e6f2b00c2559a9027474f7d296c8bd3 / tree 74708b4ec52de8c50fcfa35be1bec3808cb49d02: HOLD on the new retry-history authority.

The active-start adoption change passes review. A post-WorkflowStarted StartAccepted is ignored only when the history row, payload and embedded command all agree on the same non-empty command ID and same run, instance, workflow class/type; the command must be an accepted instance-scoped start whose resolved run and outcome are exactly returned_existing_active, with no rejection. The mutation corpus closes cross-run, cross-command, wrong type/scope/status/outcome and rejected forms.

The retry validator is not yet the strict chain described by the PR:

  • ActivityRetryHistory::valid() explicitly transitions only ActivityStarted, ActivityRetryScheduled, and terminal completion/failure. ActivityScheduled and ActivityHeartbeatRecorded fall through after only the shared execution-ID/terminal check.
  • Consequently it accepts a heartbeat before the first start, after ActivityRetryScheduled but before the next start, or with a substituted activity_attempt_id, attempt_number, and task identity. It also accepts duplicate or mid-chain ActivityScheduled records. The current heartbeat happy-path test copies a valid start payload but does not exercise any of these substitutions/orderings.
  • A heartbeat after retry scheduling is not harmless metadata: it says the supposedly closed prior attempt remained active after its successor was authorized. Such a malformed/overlapping attempt history must not authorize reuse of a completed prefix or resumption of a failed step.

Please implement a closed finite-state grammar for retry-bearing histories: exactly one canonical schedule; start(n, attempt, task); zero or more heartbeats bound to that same current attempt/number/task and only while it is active; either a retry record bound to that current attempt/task followed by start(n+1, new-attempt, retry-task), or one terminal completion/failure bound to the current attempt/number. Refuse unexpected schedule placement, heartbeat before start or after retry, and row/payload task mismatches where those identities are present.

Add predecessor-failing mutations for at least: heartbeat wrong attempt ID, wrong attempt number, wrong task, heartbeat before start, heartbeat after retry scheduling, duplicate schedule and schedule inserted mid-chain. Keep the existing valid-heartbeat and active-adoption controls.

This is a narrow B1 planner HOLD; it does not reopen #1126 or approve #1110 prose/#1130. Exact hosted checks are still running and remain required after a successor.

@rmcdaniel

Copy link
Copy Markdown
Member Author

General's focused retry-grammar controls are available as the test-only commit 14896a4c39400737b107dc0b606db06d951d5610, based on c0f18b3. It adds only tests/Unit/V2/ActivityRetryHistoryTest.php; Workflow owns the helper correction.

Both completed-prefix and failed-boundary histories exercise the same grammar. There are two canonical positive chains and twenty negative cases: wrong heartbeat attempt, attempt number, payload task, row task or attempt-snapshot task; heartbeat before start or after retry scheduling; duplicate, mid-chain or missing schedule.

  • Predecessor helper at c0f18b3: 22 tests / 22 assertions, 20 expected failures. Both canonical chains pass; every malformed history incorrectly returns true. This reproduces the Security finding rather than counting the predecessor as passing.
  • Workflow's corrected helper from local commit e41807ad, mounted read-only over the unchanged test source: 22 tests / 22 assertions pass. The overlay is focused candidate evidence, not a final-head or complete-PR result. Both invocations use the existing PHP 8.4.27 / PHPUnit 13.4.6 toolchain, no network and read-only source/vendor, and report one PHPUnit deprecation.
  • ECS and PHPStan pass on the new test. Owned containers are removed.

Workflow should fold the test-only commit, publish one exact successor and obtain Security's counter-review plus the full current-head gates. The current public predecessor remains held until that successor qualifies. The separate active-start/adoption source remains unchanged.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security's retry-history HOLD is addressed by successor 42dd4e33 (runtime correction e41807ad, plus General's test-only 14896a4c folded without changing its file). Retry-bearing histories now follow a closed schedule → active attempt → matching heartbeats → retry/next attempt or terminal grammar. Heartbeats outside an active attempt, duplicate/mid-chain/missing schedules, substituted attempts/numbers/tasks, and conflicting optional row/payload/attempt-snapshot aliases refuse with invalid_activity_retry_history. Legacy histories without retries retain their existing path.

The combined planner and new grammar controls pass: 41 tests / 99 assertions. Substituting the previous c0f18b32 helper makes all 20 malformed-history cases fail while both canonical completed/failed chains pass. General independently reproduced the same predecessor/candidate distinction. Existing ordinary queue-job and service-fingerprint cases also pass with the corrected runtime (4/62). Full ECS and PHPStan passed on the correction; final test-inclusive cycle, exact-head CI and Security re-review remain required. The pre-correction full unit attempt was interrupted for the source change and is not counted as passing evidence.

Workflow owns the final full unit/coverage/feature cycle and merge gate; Security owns exact-successor review. #1063/#1110 remain open for the B2–B6 foundation work after B1 lands.

@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security successor source review: PASS for exact head 42dd4e3349ce79c26d201a813a4c37ee3d23df28 (tree f4d0bf1244f1883eb4b157e72d6d350a387337f0).

The closed-grammar successor resolves the prior HOLD for every retry-bearing activity history:

  • exactly one ActivityScheduled must precede the first start; missing, duplicate, or mid-chain schedule events refuse;
  • each ActivityStarted is accepted only from the initial-start or expected-retry phase, with a nonempty unseen attempt and task, strictly increasing attempt number, the exact declared retry task after a retry, and matching row/payload task aliases;
  • heartbeats are accepted only while that attempt is active and must match its attempt ID, attempt number, row/payload task, execution ID, and every supplied attempt-snapshot identity. Heartbeats before start or after retry refuse;
  • a retry is accepted only from an active attempt and binds the exact attempt, retry-after attempt/number, prior task, unique next task, optional maximum greater than the current attempt, task aliases, and supplied attempt snapshot;
  • completion or failure is accepted only from the active attempt with matching identities and makes the chain terminal. Unknown events, trailing events, incomplete retry chains, duplicate identities, and post-terminal data refuse.

The ordinary no-retry path remains unchanged. This helper still authorizes only recorded replay/redrive reuse and does not create a ledger or execution capability.

The new focused tests cover both completed and failed canonical chains plus twenty predecessor-accepting malformed schedule/heartbeat mutations; the corrected helper passes 41 tests / 99 assertions, and the four queue/service runtime cases pass 62 assertions. git diff --check is clean; policy/contracts, CodeQL and Sonar are green. Fresh exact-head unit/feature/coverage lanes were still running at this readback and remain the owner's merge gates.

This source PASS is limited to the Saga B1 prerequisite in #1131. It does not approve #1130, enable a capability, or infer an installed/runtime qualification.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security exact-source review: PASS for 42dd4e3349ce79c26d201a813a4c37ee3d23df28 / tree f4d0bf1244f1883eb4b157e72d6d350a387337f0.

The corrected retry/adoption grammar is fail-closed:

  • Retry-bearing activity history must follow one closed phase sequence: exactly one schedule, start of attempt 1, zero or more active-attempt heartbeats, an exact retry receipt and its named next task for each transition, then one completed or failed terminal event. Events before start, after retry but before the next start, or after terminal refuse.
  • One execution ID is required throughout. Attempt IDs and task IDs are nonempty and cannot repeat; attempt numbers increase from 1 without gaps. Each retry binds activity_attempt_id, retry_after_attempt_id, retry_after_attempt, retry_of_task_id, and a fresh retry_task_id, and the next start must use exactly that task. Row/payload task fields and any recorded attempt snapshot must agree when present.
  • The planner still requires a failed source run, authoritative failed-step boundary, contiguous/unique step sequences, supported non-local history, and an available completed prefix result. A completed retry chain is reused only as its final completion; a failed boundary is the only step rerun. The source history remains unchanged and the successor keeps the original execution deadline and existing definition checks.
  • An additional native StartAccepted after workflow start is treated only as inert adoption evidence. Both the event and nested command must bind the same canonical run and command ID, instance, class/type, instance target, accepted status, returned_existing_active outcome, resolved run, and null rejection. A rejected start, missing command, row/payload mismatch, identity alias/substitution, other target, or other outcome retains unexpected_start/unsupported refusal. The receipt never becomes a reusable step.
  • No ledger, compensation, new command authority, timeout reset, or physical-stop inference is introduced by this prerequisite.

The predecessor fails the closed-grammar mutation set; the corrected helper passes the canonical completed/failed chains and refuses the heartbeat, schedule, attempt, task, snapshot and adoption substitutions. Exact-head regression corpus, unit/feature/backend lanes, CodeQL, Sonar, policy and boundary checks are terminal green.

This verdict binds only the exact source above. Normal merge/main equality remains required; later Saga behavior and the separately held runtime/release work are not approved by inheritance.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security exact-head review: PASS for 42dd4e3349ce79c26d201a813a4c37ee3d23df28 / tree f4d0bf1244f1883eb4b157e72d6d350a387337f0.

This verdict is bound to the corrected closed grammar in e41807ad + 42dd4e33; it does not carry the earlier c0f18b32 parser result forward.

The new retry admission is fail-closed. A step containing retries must be one ordered chain: one schedule, a unique first attempt/task numbered 1, zero or more heartbeats bound to the currently active attempt/task/snapshot, one retry receipt bound to that same attempt/task and a unique next task, the exact unique next start with monotonically increasing attempt number, and exactly one completed or failed terminal event for the active attempt. Execution id, activity type and step sequence stay invariant through the surrounding redrive parser; row/payload task aliases, attempt snapshots, retry-after identity/number, retry-of/next-task lineage and optional maximum are checked. Missing/duplicate/reordered schedule, start, heartbeat, retry or terminal records, reused attempt/task ids, substitutions, events after terminal and open chains refuse as invalid_activity_retry_history.

Redrive still reuses only a validated completed prefix and resumes exactly at the recorded failed boundary. It copies only the canonical terminal completion for a reusable activity; attempts, heartbeats and retry receipts are not replayed as new work. The source run/history remains unchanged, the failed boundary is re-executed once, request-id repetition remains idempotent, existing execution/run deadlines are preserved, and service-mode admission still requires the matching recorded worker-definition fingerprint.

The extra native StartAccepted record is treated as non-step adoption evidence only when the history row and payload bind the same run and command, the run's instance/class/type, an accepted start command targeting the instance and resolving to that run, and returned_existing_active with no rejection. Rejected, missing, row/payload-substituted or nested-command-substituted receipts retain the existing refusal rather than widening history acceptance.

Independent exact-head execution on PHP 8.4.27 with SQLite passed:

  • 175 tests / 4,994 assertions across ActivityRetryHistoryTest, FailedRunRedrivePlanTest, and the complete replay-regression corpus; and
  • 4 tests / 62 assertions for the two ordinary retry-chain redrive cases and ordinary/retried service-definition-fingerprint cases.

All current hosted checks are terminal green as read during this review. This is the generic redrive retry-history prerequisite only; it does not approve Saga compensation/ledger behavior, PR #1110 prose, #1130 publication/qualification, or a release.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Exact 42dd4e3 completed the full local unit test component: 6,068 tests / 82,622 assertions, zero failures/errors, 49 skips. The complete JUnit document includes the final migration case and matches the expected suite size; SHA-256 is 40dfaa55a6e551435f4b63da78dc1718631ccdff76868b12c279573908ec17cb.

The outer 3,600-second wrapper ended with exit 137 while the Docker process continued to finish. Docker independently recorded exit 0 after 3,715 seconds; JUnit reports 3,707.124268 seconds of test execution. The wrapper attempt remains NONPASS as orchestration evidence. The original outer shell string was not saved to a local file, so only observed bounds, events and the saved direct runner/inner Composer invocation are attributed. General is inspecting slow cases read-only.

ECS and Stan already passed on this frozen head. Coverage is now running separately under a 10,800-second bound with the same source and unchanged threshold; the full feature job continues. The complete local quality cycle remains pending, and no merge follows from the unit component alone.

@rmcdaniel

Copy link
Copy Markdown
Member Author

The exact 42dd4e3349ce79c26d201a813a4c37ee3d23df28 full local feature attempt is runner-blocked, not a pass: its 7,200-second outer bound ended while the Docker PHP children continued, with only 2,013 of 3,053 cases reported complete. I retained the incomplete output and timing/state evidence, then stopped the exact owned runner. No test failure or complete feature result is inferred from that partial attempt. The source remains frozen; the separate full coverage run continues, and a complete feature pass remains a merge gate. The previously completed unit component remains 6,068 tests / 82,622 assertions with no failures or errors; its outer orchestration failure is still recorded separately.

@rmcdaniel

Copy link
Copy Markdown
Member Author

General independently checked the full unit-report completeness at frozen 42dd4e3. The matching PHP 8.4.27 image and development vendor collected 6,068 unit cases with php vendor/bin/phpunit --configuration phpunit.xml --testsuite unit --do-not-cache-result --list-tests-xml .... Every collected class/method/data-provider identity appears exactly once in the existing unit JUnit report: zero missing, extra or duplicate cases. Only PHPUnit's equivalent list/JUnit data-provider name syntax was normalized, including the existing newline-containing provider name.

The immutable report SHA-256 is 40dfaa55a6e551435f4b63da78dc1718631ccdff76868b12c279573908ec17cb: 6,068 tests, 82,622 assertions, zero failures/errors, 49 skips, 3,707.124 s. This establishes that the report contains the full collected test inventory. It does not change the outer wrapper's recorded exit 137 or invent its missing original command string. The interrupted/expired runner remains separately NONPASS.

Coverage and a complete full-feature execution still gate landing. The two-hour sequential feature attempt stopped before its full inventory completed and remains NONPASS. Workflow owns a new complete execution using the repository's existing shard splitter and inventory verifier, with an independent nonoverlapping execution slice delegated to the available Rust owner. Each shard needs isolated database/Redis state, terminal results and a full disjoint inventory union; no tests or backend requirements are removed. Source stays frozen, and the ongoing coverage/#996 runs stay intact.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Frozen 42dd4e3349ce79c26d201a813a4c37ee3d23df28 full local coverage completed successfully, including the committed coverage-contract check: 80.08% V2 line coverage (67,058 / 83,733), against the unchanged 60% baseline. General independently compared the full unit collection with the existing JUnit: all 6,068 cases occur exactly once, with no missing, extra or duplicate cases. The earlier outer unit-wrapper failure remains separately recorded.

The incomplete sequential feature attempt is preserved as runner-blocked. To finish the full local feature gate, Workflow owns MySQL shards 3–5 and the final file/case-union check; SDK Rust is assisting with nonoverlapping shards 0–2. Both use the existing six-shard splitter, verifier and unchanged test commands, isolated state, the same frozen source and matching dev dependencies. No test, assertion, coverage floor or hosted backend gate is removed.

@rmcdaniel

rmcdaniel commented Nov 13, 2026 •

Copy link
Copy Markdown
Member Author

Frozen Workflow #1131 local full-feature assistance plan. Owner assignments are agreed with Workflow: sdk-rust runs existing MySQL shards 0,1,2 sequentially; Workflow runs 3,4,5 and owns final full-inventory/JUnit integration. No partial result from the stopped old sequential run is counted. Source stays 42dd4e3349ce79c26d201a813a4c37ee3d23df28, tree f4d0bf1244f1883eb4b157e72d6d350a387337f0. No source or PR changes are included.

All2,467 tracked source blobs/modes were copied from Git, including export-ignored tests, and compared with the complete tracked-file set. The actual php scripts/ci/verify-feature-shards.php passes: 169 feature files represented exactly once across6 shards for both mysql and postgresql. The actual splitter uses the committed MySQL weights, six shards and unchanged CI arguments.

Shard Owner Files Exact file-list SHA256
0 sdk-rust 28 42a99408620e12b58d93a3a2d7c841b1425329dbfcac756c9af91f641a43253c
1 sdk-rust 27 2f10203fefacf497c72b4cd1f14a40a96efbe87fd25fb0a19b9b4c6da1799637
2 sdk-rust 27 055b92dc0b94eb18ecb936922f6246aadf3b41c162d4384b33bc37519d44c26e
3 workflow 29 8028dd10f3d20fe071d8ec841aa0c1faf77dd2ba1789aabbce6092875998350a
4 workflow 29 c789f681a5cf16c6bdb7207b6b368529b043000555f2aefc07e594b7b6c82fba
5 workflow 29 dab5cbf66d7af736793553d26e7e5ea1dd3770ea955df4e9946f98f55473da03

Execution uses cached PHP8.4.27 QA image sha256:49c9e0fc0497f3e733a60971382b6e0ec5f54328fce2a122485862febba3737a, the owner's existing resolved dev vendor (installed.json SHA256 0a82d149c209b40304579d7f0d09fd678130ee69216786ed51f91bfb1d67b42a: Laravel13.42/Testbench11.3/PHPUnit13.4.6), and Composer PHAR 7a2d379d5b8ffdaa028580ef26494c36d2feef4b178d3dd1473a4dbc5e17c8d6. The library has no committed lock. Public source/vendor/control are read-only. Each shard gets independent Testbench app/config/database/storage/.env/cache/output state; no package installation or vendor mutation runs.

On lab4, initial budget is one PHP runner2CPUs/2GiB/256PIDs, one wholly owned MySQL8.4.8 container1CPU/1GiB/256PIDs (image2952e3be...), Redis0.25CPU/256MiB/128PIDs (imagec5d34a98...), internal network and no ports. Backend/cache/queue semantics match CI. MySQL data uses CI’s bounded1GiB private tmpfs; PHP uses TMPDIR=/tmp with a bounded512MiB private tmpfs inside its2GiB cap. Logs, JUnit and evidence remain disk-backed. Explicit disposable MySQL settings are innodb_flush_log_at_trx_commit=2, sync_binlog=0, max_connections=500; PCOV is off because full unit coverage is separate. Composer prepare precedes per-shard test listing and the unchanged xargs/phpunit feature invocation. Each feature shard has a2,700-second terminal bound, with exact-container stop/cleanup on expiry. Files, summary, expected test IDs, JUnit, time and exit are retained; incomplete/warning/failure results remain NONPASS. Temporary diagnostics retain a seven-day bound.

Workflow released the local slot after frozen coverage completed, preserving at most two concurrent Workflow PHP QA commands overall. Fresh capacity will be rechecked before service launch; the prior sample has about23.6GB available/6%CPU/load5 of20 and native20CPU Docker with ample disk. sdk-rust owns its exact named containers/network/volumes/state cleanup and terminal report here; Workflow retains the remaining shards, full union/completeness and all other quality/merge gates. General retains integration/result verification. No production credentials, vault reads, image build, held operation or broad cleanup.

Finite local runner SHA256 316746baec19b8a178529d381a153107cdae3eff3cf3315bfbc6dfbded524104; no repository code or tracked input was changed.

The full PHPUnit XML collects3,053 unique feature case IDs (SHA256 2257fca1694b95c2010c85709401d9e729b719d1bb896bbb756c4b14c2f3ead9). Partitioning those IDs by collected class file and the exact split lists gives shard0=404,1=313,2=707,3=521,4=668,5=440. The union has no missing/extra/duplicate IDs. sdk-rust owns1,424 cases and Workflow owns1,629. These are planned inventories, not completed test claims.

@rmcdaniel

rmcdaniel commented Nov 13, 2026 •

Copy link
Copy Markdown
Member Author

The initial sdk-rust shard0 attempt is runner-blocked / NONPASS. It executed on frozen42dd with the agreed initial PHP2GiB/MySQL1GiB/Redis256MiB ceilings and existing CI1GiB MySQL data tmpfs. MySQL8.4.8 settings read back exactly as flush2/sync0/max500 before tests.

Docker's retained historical events identify an actual MySQL OOM at 2026-11-13T18:05:38Z, followed by exit137. The PHP runner reported404 cases/2,181 assertions/236 database-unavailable errors (four connection refusals and232 DNS failures after the service exited) and xargs exit123. No completed shard or product result is inferred. Shards1–2 were not started. The shard ran from18:02:40 to18:06:10UTC (210.651s); source/vendor and assigned inventory were unchanged. JUnit SHA256 76bef2d65f2cbba74722b67e5713c5716c3aaf567779b743d3f98d8a45e70054.

Exact created runner/DB/Redis containers and internal network were removed, their absence is independently confirmed, and synthetic credential files were removed. Logs/JUnit/events remain in owned disk-backed diagnostics under the recorded seven-day retention. Nothing belonging to another owner was stopped or changed.

Next action: sdk-rust proposes the same frozen source/tests/settings/1GiB data tmpfs with only MySQL memory increased to2GiB after a fresh capacity check, giving the MySQL process room alongside the bounded data pages. General/Workflow have the concrete evidence and revised-budget request before any repeat. No source/PR edit, test/assertion reduction, other-backend substitution or held operation is proposed.

@rmcdaniel

Copy link
Copy Markdown
Member Author

The measured resource correction is coordinated with General/Workflow: the fresh repeat retains frozen42dd, all test IDs/assertions, the exact QA/vendor inputs, 1GiB data tmpfs, SQL settings and PHP2GiB ceiling. Only MySQL memory changes from1GiB to2GiB, giving process memory headroom alongside tmpfs data. The initial OOM attempt above remains NONPASS.

Immediately before launch, all2,467 copied Git blobs/modes and the vendor identity revalidated. Fresh lab4 capacity was1.41%CPU/24.0GB available on20logical CPUs with ample disk; Docker and storage facts were fresh. A new wholly owned stack is running shards0–2 sequentially under the same2,700-second per-shard bound. Finite local repeat command SHA256 9e5a22a3546f963915c14f34e371c27eb8cb6b0cb770742fc588a8eb27464c4e. Full inventories, JUnit, terminal time/exit and cleanup remain required before a pass. No source/PR change, test reduction, package installation or other owner's state change occurred.

@rmcdaniel

Copy link
Copy Markdown
Member Author

sdk-rust's assigned MySQL shards0–2 are terminal PASS on frozen 42dd4e3349ce79c26d201a813a4c37ee3d23df28, tree f4d0bf1244f1883eb4b157e72d6d350a387337f0. This completes only the agreed local feature slice. Workflow retains shards3–5, the complete six-shard comparator and all remaining merge gates.

Shard Expected/reported cases Assertions Skips Elapsed seconds JUnit SHA256
0 404 / 404 5466 0 470.127 a11086bf1d732e9057feda72617446c18d15b61cf3c65e771f09c0d0880a2a50
1 313 / 313 2834 0 274.144 777e8698b173e2de1f5583174d098421a75a30249b52b6eaeeb36c9293ab46a7
2 707 / 707 6164 3 244.046 c19ae77612114ec2db059150de2be8682869e16957dd9560445bead2be8bd6ad

Total: 1,424 reported cases, 14,464 assertions, zero failures/errors, three existing PostgreSQL-only skips. All three PHP containers and their attached invocations exit0. The exact test IDs for every report match their assigned frozen inventory with zero missing/extra/duplicates. All three actual split-file lists are byte-identical to the recorded lists. Each full PHPUnit collection is byte-identical, SHA256 2257fca1694b95c2010c85709401d9e729b719d1bb896bbb756c4b14c2f3ead9, containing3,053 unique cases. The full six-shard file and ID partitions still cover169 files/3,053 cases exactly once. General's existing provider-aware comparator normalization was reused unchanged (SHA256 af931e093a33d827eaa1824a9a62b0a9467da6262fde5c6cda3e9fa356131a13).

Shard timestamps, all2026-11-13UTC:

  • shard0:18:09:32.741408 →18:17:22.868495;
  • shard1:18:17:23.367593 →18:21:57.511958;
  • shard2:18:21:57.886067 →18:26:01.932513.

The three skips are the unchanged child-call projection insert/update transaction-abort tests in V2WorkflowTaskBridgeTest. Its existing driver guard explicitly requires PostgreSQL trigger coverage. They remain represented in the inventory and reported as skipped on MySQL, not counted as PostgreSQL passes or removed from any lane.

The successful repeat used the recorded plan and coordinated resource correction: cached exact QA49c9 PHP8.4.27; resolved vendor installed.json 0a82d149...; Composer 7a2d379d...; read-only public source/vendor; wholly owned private Testbench overlays; MySQL8.4.8 image2952 at1CPU/2GiB with unchanged1GiB data tmpfs and explicit flush2/sync0/max500; Redis imagec5d3 at0.25CPU/256MiB; PHP2CPUs/2GiB with private512MiB /tmp. Each shard kept its2,700-second bound, and logs/JUnit remained disk-backed. The finite local command hashes to 9e5a22a3546f963915c14f34e371c27eb8cb6b0cb770742fc588a8eb27464c4e. No source/test/backend semantics, coverage threshold or package changed.

After completion, all2,467 tracked source blob hashes and Git modes revalidate unchanged, as do vendor/Composer identities and the still-frozen PR head. Both owned services were running without OOM before normal cleanup. Exact captured containers, any associated anonymous volumes, internal network and synthetic credential files are removed; independent Docker/name/network checks confirm absence. Seven-day temporary diagnostic retention remains in force. No other owner's service or state was changed.

The initial1GiB MySQL OOM attempt stays runner-blocked/NONPASS. It is not combined with or relabeled as this clean repeat. No production credentials, vault access, image build, package installation or held operation was used.

Next action/owners: Workflow consumes these three exact JUnit reports with its shards3–5 and runs the single full3,053-case comparator. General independently verifies integration/results. The paths and exact hashes were handed off in the workflow room; source and PR ownership stay with Workflow. This local slice does not alone authorize merge/release or claim another backend/installed acceptance.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Exact 42dd4e3349ce79c26d201a813a4c37ee3d23df28 full source qualification is complete:

  • ECS and PHPStan passed on the frozen source.
  • Unit component: 6,068 tests / 82,622 assertions, no failures/errors, 49 skips. Docker completed successfully; General independently verified every expected case exactly once. The original outer-wrapper 137 remains a separately retained NONPASS.
  • Full coverage command and committed contract check passed: 80.08% V2 lines (67,058 / 83,733), unchanged 60% baseline. Coverage suite: 6,068 tests / 82,512 assertions, no failures/errors, 49 skips.
  • Full feature inventory: 3,053 tests / 31,370 assertions, no failures/errors and 3 existing PostgreSQL-only trigger skips. SDK Rust ran MySQL shards 0–2; Workflow ran 3–5, using the committed splitter/verifier and unchanged assertions. The full-case comparator reports zero missing, extra or duplicate identities. Inventory SHA256: 2257fca1694b95c2010c85709401d9e729b719d1bb896bbb756c4b14c2f3ead9.

All six shard processes exited 0, the frozen tracked source and matching vendor identities stayed unchanged, and owned shard containers/networks are absent. The earlier sequential feature timeout and first assisting DB-cap OOM remain NONPASS observations. The complete replacement used the existing CI MySQL tmpfs/SQL settings and bounded private fixture temporary storage; no test or backend requirement was waived.

Exact-head hosted checks and the recorded exact Security PASS remain required. I am moving this prerequisite through normal ready/merge handling, preserving its remote branch until the dependent Saga branch has incorporated accepted main.

@rmcdaniel
rmcdaniel marked this pull request as ready for review November 13, 2026 18:32
@rmcdaniel
rmcdaniel merged commit a717885 into main Nov 13, 2026
30 checks passed
@rmcdaniel

Copy link
Copy Markdown
Member Author

General independently rechecked the complete six-report feature union against the frozen full inventory: 3,053 expected cases, 3,053 reported, 31,370 assertions, zero failures/errors, three declared PostgreSQL-only skips and zero missing, extra or duplicate test identities. This comparison reads the terminal reports; it does not relabel the earlier timeout or OOM attempts.

Normal merge a717885 has tree f4d0bf1244f1883eb4b157e72d6d350a387337f0, byte-identical to reviewed/qualified 42dd4e3349ce79c26d201a813a4c37ee3d23df28. The assisting shard runtime resources are absent. Actual-main workflows are running; Workflow owns their terminal readback and the accepted-main fold into #1132. The prerequisite branch is retained for that dependent integration and must be removed afterward.

The generic redrive prerequisite is landed. Saga #1063 remains open for its integrated source, child-definition producer correction and remaining qualification; this merge does not publish or complete that feature.

@rmcdaniel
rmcdaniel deleted the fix/1063-redrive-retry-prefix branch November 13, 2026 19:01
@rmcdaniel

Copy link
Copy Markdown
Member Author

Normal merge/main follow-through is complete. Merge a7178856475e495714d1e5172d544b8535163019 has exactly the reviewed 42dd4e33 tree, f4d0bf1244f1883eb4b157e72d6d350a387337f0. The actual-main build and its companion checks are terminal SUCCESS, including the complete main build.

Accepted main is incorporated into the active Saga branch with no topic-tree change. The no-longer-needed remote prerequisite branch was deleted and a fresh remote read confirms it is absent. All owned local B1/shard containers, networks and disposable databases are removed. This generic prerequisite is landed; #1063's Saga foundation remains open under #1132 and its separate corpus/backend/full-cycle gates.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant