Skip to content

Settle the pending payload deletion before asserting the bytes are gone (#970) - #997

Merged
rmcdaniel merged 1 commit into
mainfrom
test-970-stream-payload-race
Nov 3, 2026
Merged

rmcdaniel merged 1 commit into
mainfrom
test-970-stream-payload-race

Conversation

@rmcdaniel

Copy link
Copy Markdown
Member

Review tier C: test only. It changes one PHPUnit helper and no application code.

Fixes #970.

Cause

WorkflowStreamPayloadRaceTest::test_a_writer_and_payload_cleanup_decide_a_shared_object_under_its_lock failed three times on PRs that don't touch PHP: #943, #965 and #995. Each failure was a SQLite cleanup-first case with the same assertion: the shared object's file still existed. Before that assertion, cleanup had reported the object deleted and the writer had been refused.

Cleanup doesn't delete the bytes inside its transaction. RuntimeExternalPayloadDeletions::stage commits a durable deletion intent together with removing the last registration, and the provider deletion runs in an after-commit flushPending. If that step fails, it catches the error and keeps the intent for the next cleanup pass (retryPending), by design.

In these cases the test releases the writer just as cleanup commits. On SQLite the writer's lock-taking write can commit between flushPending's read and its write, which makes the write fail at once with a busy-snapshot error. The bytes then wait for the next pass, but the test asserted they were already gone. I haven't observed the busy error directly in CI; the fault injection below reproduces the exact symptom through that path.

Change

assertObjectDeleted now allows the bytes to outlive the commit only when exactly one deletion intent for that URI is pending. It then runs retryPending('default', 10), as ExternalPayloadRetentionRollbackTest already does, and requires:

  • the bytes gone,
  • no intent left,
  • no registration.

This is stricter than before in one respect: the intent row must also be cleared.

Evidence (lab2, the repo's base image, SQLite data sets)

  • Fault injection: I made the cleanup actor's after-commit flushPending fail once, in the container's copy only. On the sqlite: http append, run prune first data set:
    • main's helper fails with the CI message: Failed asserting that file ".../avro/3c/3ccdc67e…" does not exist;
    • this helper passes by settling the pending intent.
  • Without injection: the file's 13 SQLite tests passed 8 times in a row.

🤖 Generated with Claude Code

…ne (#970)

Cleanup commits the object's last registration together with a durable
deletion intent, and deletes the bytes after that commit. On SQLite the
after-commit step can lose the write lock to the writer the test has
just released; it then keeps the intent for the next cleanup pass, as
designed. The test asserted the bytes were gone immediately, so it
failed intermittently on unrelated PRs.

Bytes may now outlive the commit only with exactly that intent pending.
The test runs the retry pass, as the other retention tests do, then
requires both the bytes and the intent to be gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sonarqubecloud

sonarqubecloud Bot commented Nov 3, 2026

Copy link
Copy Markdown

@rmcdaniel

Copy link
Copy Markdown
Member Author

Tier C receipt. Hosted checks are terminal at exact head 5ab3a35c43ab2d8cd2540f54eb2379afbb14bf8a (tree 1ed7458ae46f720c01702eb61668f0ec92ee215f): 15 passed, 8 skipped, 0 failed. PHPUnit feature suite and Feature source qualification passed, including every WorkflowStreamPayloadRaceTest data set on all drivers. The skipped jobs are the native and fixture builds, which this PHP-test-only change doesn't trigger. One file changed: tests/Feature/WorkflowStreamPayloadRaceTest.php.

@rmcdaniel
rmcdaniel marked this pull request as ready for review November 3, 2026 18:38
@rmcdaniel
rmcdaniel added this pull request to the merge queue Nov 3, 2026
Merged via the queue into main with commit 192acf3 Nov 3, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: WorkflowStreamPayloadRaceTest leaves the shared object on SQLite (two data sets, unrelated PRs)

1 participant