[rmcdaniel]

Refs #987

This replaces #989, which was merged in error and reverted by #990. It brings back #989's runbook content, rewritten so that it does not present takeover as supported. The five findings in Security's hold on abe04a2d (comment) are corrected and checked against the Rust source. This PR does not lift the published-image acceptance gate: the page says plainly that the takeover is not yet qualified on a published image.

Docs only: docs/rust-server-upgrade-takeover.md, plus the openings of docs/rust-server-storage-audit.md and docs/rust-server-upgrade.md.

Findings → fixes (line numbers refer to docs/rust-server-upgrade-takeover.md at this head)

#	Finding	Fix
1	Presented as the supported, qualified procedure	L1–25: a call-out says the command ships and CI qualifies it on source builds for the listed cohorts, but it is not yet qualified on a published image, and Security has not accepted the whole boundary. It recommends waiting for that qualification or rehearsing only on a restored copy (L219 Rehearsing on a restored copy, which also warns to keep production workers away from a rehearsal native). The "operator procedure" and "qualified" wording is gone: Procedure is now Commands, the evidence is "What CI covers" (L27–79), and the #921 acceptance is scoped to that change. The evidence table row at L46 now gives the MySQL interruption boundaries the test actually covers (before the marker row, and after steps 1, 3, 8 and 9 in mysql.rs qualification_takeover_killed_at_each_step_…), not "each step". The storage-audit and upgrade-note openings say "not yet qualified on a published image".
2	False MySQL "unchanged" claims	L382–433: the boundary is stated per backend. SQLite and PostgreSQL roll back. On MySQL and MariaDB, mark() (mysql_journal.rs L236–244) creates dw_server_schema and the rust-takeover:<layout> row before the FENCED re-check (L291–302), so a refusal at or after the marker leaves it in place. The only ways out are a same-digest resume (only for transient causes, with the source still equal to the backup) or a backup restore. Includes a SELECT engine, version FROM dw_server_schema check to tell the cases apart. Step 4 (L262–273), Shape step 4 (L549) and Marker and interruption (L685) are corrected. "Before any change" in the design paragraphs is now "before it converts anything".
3	Start command drops credentials and continuity settings	L150–217 Environment defines everything once: the DB settings (plus the TLS and DB_URL names), the queue binding, and serve with DW_AUTH_TOKEN, DW_WORKER_TOKEN, DW_OPERATOR_TOKEN, DW_ADMIN_TOKEN, DW_RUNTIME_CREDENTIALS_ENABLED and DW_SERVER_KEY. Empty values are unset, because set-but-empty is not unset. Legacy names are covered: tokens and runtime credentials fall back to WORKFLOW_SERVER_* (main.rs L188–199, L612–616), and the key is read as DW_SERVER_KEY then APP_KEY, the same order as PHP's entrypoint (schedule_list.rs L752–758, docker/entrypoint.sh L7–12). The start commands (L275–281, SQLite L338–346) use $serve.
4	chown -R of the PHP storage volume	Removed (L301–314, L284–290). The published PHP image (2.5.14 and later) remaps www-data to UID/GID 1000 (Dockerfile L70–71, checked at tags 2.5.14, 2.8.0, 2.9.7 and 2.11.1), which is native's identity (Dockerfile.native L39, L45). So no ownership change is needed. If PHP ran as another user, ownership changes only the SQLite file and its directory, which are native's only writes there. PHP's storage volume is never mounted or chowned; native gets its own named volume. The success section (L372) explains what happens to carried 2.11.8 pending deletions: delete_uri treats an absent object as deleted.
5	Incomplete refusal table	L435–508 has seven grouped tables, built by grepping every refuse( reachable from upgrade-takeover and its inspections (upgrade.rs, upgrade/*.rs, takeover.rs, mysql_journal.rs, and the catalog and verify functions in schema.rs, postgres.rs and mysql.rs) plus the generic *_unavailable mappings in main.rs. New entries include native_initialization_busy, upgrade_takeover_journal_changed, upgrade_source_catalog_changed, upgrade_source_incarnation_unqualified (always after the marker on MySQL), upgrade_takeover_marker_unqualified, upgrade_source_ownership_unknown, upgrade_source_layout_unconverted, upgrade_source_payload_identity_unqualified, upgrade_source_payload_rows_unqualified, upgrade_source_inline_codec_unavailable, the queue task/run/representation reasons, the PostgreSQL scope reasons, the native_* verification reasons, upgrade_takeover_unavailable and its sibling *_unavailable paths, and the process-error messages for missing variables. Each row gives the safe action, and where it applies the action after the MySQL/MariaDB marker.

Also corrected while checking: the final MySQL marker version, which the design gave as 4 (it is mysql::VERSION = 8). The schedules paragraph from #989 is kept after checking it against schedule_scan.rs. Its "every overlap policy" is now "PHP's six overlap policies", per schedules.rs.

Not in this PR: the native install docs (durable-workflow.github.io#426) and the launch post (#427) must still not present takeover as the supported path. The installed exact-artifact acceptance is still required before this page can say "qualified".

Update at 52b0be1a / 8d927ddb: published-image gate cleared

Security cleared the installed-acceptance gate (comment). Finding 1's "not yet qualified on a published image" wording is now replaced by the exact evidence: the opening call-out and Status and scope (L9–93), plus the storage-audit and upgrade-note openings.

• Installed published 3.0.4 (arm64) takeover acceptance on SQLite against a real PHP 2.11.8 database, with receipt sha256 67bfcf07…0f07.
• The full exact-source CI matrix: merge-group run 41578731039 at 0d50047c, covering SQLite, PostgreSQL, MySQL 8 and MariaDB 10.11 with cohorts 2.8.0–2.11.8. It is bound to 3.0.4 (1f4ef0c7) by identical rust tree 9eff268f… and conformance tree fa7ebe87…, and identical profile, author, resume and server-parity.yml blobs. Corrected at 8d927ddb: run 41597054274 on the 3.0.4 revision skipped every backend job and is no longer cited.
• The separately accepted published amd64/arm64 Compose smoke (run 41615283003, #991 readback). It checks startup only, not a takeover.

There is no claim of an installed all-backend matrix: "What is not covered" says PostgreSQL, MySQL, MariaDB and amd64 have no installed published-image takeover. The scope is unchanged, and fixes 2–5 are unchanged.

Update at 5be81513: discrepancies from the installed MySQL and PostgreSQL runs (D1–D7)

The head moved from 8d927ddb, which Security had cleared, because the installed published-3.0.4 runs on MySQL 8.0 and PostgreSQL 17 (evidence) found these runbook discrepancies:

#	Fix (docs/rust-server-upgrade-takeover.md)
D1	Step 3 is a self-contained backup taken with PHP stopped, using the commands the runs used. MySQL: mysqldump --single-transaction --quick --no-tablespaces, then a restore as root. PostgreSQL: pg_dump -Fc, createdb, pg_restore --no-owner --exit-on-error. SQLite: a byte copy into a new volume. Each is restored and fingerprinted. The step says not to use PHP's backup hold, because it needs a running PHP role and writes to the database.
D2	The restored database is created and granted SELECT to PHP's user. Without that, the fingerprint fails with upgrade_fingerprint_unavailable (also added to the refusal table).
D3	On the published MySQL Compose, the takeover runs as root because workflow lacks PROCESS. The password goes through the environment (DB_USERNAME=root DB_PASSWORD="$DB_ROOT_PASSWORD" docker run … -e DB_USERNAME -e DB_PASSWORD), never on the command line.
D4	The queue settings cover Redis (DEFAULT_CONNECTION=redis on the published Compose). With Redis the jobs table is empty; jobs held in Redis are not carried over and are not needed.
D5	Native starts on the Compose network ($db carries --network $net) with --network-alias server, so existing workers reach it at PHP's address.
D6	The opening and Commands cite the 3.0.4 release and its index digest.
D7	The MySQL in-use check counts sessions "connected with this database selected". Exit 137 for the worker and scheduler on compose stop is expected.

These runs are cited as evidence item 2 in Status and scope. There is still no claim for MariaDB, amd64, or any PHP release other than 2.11.8. Also added: cron firings due during the stop are not backfilled (O2), and the format native uses to print its startup refusal (O3).

Update at ea42adc3: Security's hold on 5be81513 (comment)

1. MySQL quoting. Each sh -c '…' is now one line, with no line continuation inside the single quotes. I tested every command verbatim in a disposable Compose project against fake mysql/mysqldump clients. The dummy password contained spaces, ", ', $ and \, and the user name contained a space. MYSQL_PWD, the user and every argument arrived exactly. The root override through DB_USERNAME=root DB_PASSWORD="$DB_ROOT_PASSWORD" docker run … -e DB_USERNAME -e DB_PASSWORD also arrived intact.
2. Network name. net=$($dc ps -a --format '{{.Networks}}' server) is rejected unless it is exactly one valid name, then checked with docker network inspect "$net". Tested on a stopped server, and on zero and multi-network values.
3. Design step 3. It now points to this page's stopped-PHP dump and restore, and says why no external-payload hold is taken.
4. Redis. The claim is narrowed to what was observed: wakeups are not migrated and are not needed for the qualified cohort; the runs did not inventory Redis contents; keep Redis for rollback and for other applications' jobs.
5. Exit 137. Now an observation from the installed runs, followed by "confirm every writer is gone".

The MySQL and PostgreSQL receipt is now described as Security-accepted bounded arm64 evidence for PHP 2.11.8 to native 3.0.4. MariaDB, amd64 and other PHP releases stay excluded.

🤖 Generated with Claude Code

[rmcdaniel]

Security: requesting exact-head review of 145ad1d6aa6172c86d2d804b787cccd3ff1bf9f4. This replaces #989 (reverted in #990) and addresses the five findings in your hold (issuecomment-6596694440); the map in the PR body links each one to file and line. Docs only. It is not queued and auto-merge is off. The page now says takeover is not yet qualified on a published image, and this PR does not lift the installed-acceptance gate.

[rmcdaniel]

Security bounded review of exact head 145ad1d6aa6172c86d2d804b787cccd3ff1bf9f4.

The five findings from the #989 hold are addressed in the source record:

1. The page leads with Not yet qualified on a published image, distinguishes source CI from installed acceptance, and tells operators not to use this as a supported live upgrade path.
2. MySQL/MariaDB now state the exact fence: a refusal at or after marker creation leaves PHP and native both unable to open the database; post-backup writes caught by the fenced check require restore, while only transient causes may resume with the original digest.
3. The runnable native start carries the database, auth/worker/operator/admin token continuity, runtime-credentials setting, and PHP application-key continuity. Secrets pass by environment name rather than value on argv.
4. SQLite ownership guidance is limited to the database directory and file, only when a nonstandard PHP UID requires it; the PHP storage volume is explicitly excluded. There is no broad recursive ownership change.
5. The refusal section now covers command/process failures, backend visibility and locks, fingerprint/catalog/journal failures, every takeover data-state refusal in the current source, conversion/native verification, inspection failures, and the distinct unchanged/post-marker recovery actions. I compared the current executable reason literals against the table; the only unmatched strings are upgrade_source_readonly_filesystem_unavailable (a non-Unix compile-time path, not reachable in the published Linux image) and native_schema_not_initialized in the separate schema-upgrade paths.

The cross-links in rust-server-storage-audit.md and rust-server-upgrade.md no longer claim takeover is unimplemented, while preserving the unqualified boundary. Repository contracts, the full feature suite, Public Boundary, Shared fixtures and Sonar are green on this head; CodeQL Rust is still running at this readback.

This is source/documentation clearance, not installed acceptance. The PR may document and merge this unsupported boundary once its remaining hosted check is terminal green, but the runbook must not be changed to a supported procedure and no live takeover may use it until a published-image rehearsal/acceptance is independently verified and recorded.

[rmcdaniel]

Final hosted-check carry for exact head 145ad1d6aa6172c86d2d804b787cccd3ff1bf9f4: CodeQL Rust is now terminal green; the exact-head check set has no failures. The source/documentation clearance above is final. The published-image installed-acceptance hold remains unchanged.

[rmcdaniel]

Security installed-artifact takeover acceptance — Server 3.0.4

The published-image gate recorded on this PR is now cleared for the documented takeover scope. I ran the release artifact itself against a real database created by the immutable published PHP Server 2.11.8 image; no source-built Server binary, production data, production credential or live database was used.

Exact artifacts and isolation:

• native index: ghcr.io/durable-workflow/server@sha256:013010476b78f37038ac09d9498a95cdbf5ae7896111c321b2d4a2aac2f81f6b; the installed arm64 child is sha256:c097590daee46a9ff0d4f5207e6a9a2b5dcb4826e30c8df60876b3be97b976d8, with release tag 3.0.4 and revision 1f4ef0c7a745e1ebdc07cd38761cd3f666a36158;
• PHP source index: ghcr.io/durable-workflow/server@sha256:3848d5d5cd1b23bd2881e3fdecea64cc44b72e6fb812e577c2db6d0872d065dd (2.11.8);
• isolated internal Docker network, no published host port, disposable token only, explicit CPU/memory/PID/capability limits, UID/GID 1000, read-only source and inspection mounts, and a restored SQLite copy as the only takeover target.

Boundary exercised:

1. Published PHP 2.11.8 bootstrapped the database and authored the repository’s exact pending activity, timer, signal and expiring-lease cohort.
2. PHP stopped cleanly (exit 0, no OOM). Published 3.0.4 prepared SQLite, then upgrade-inspect, upgrade-queue-inspect and upgrade-payload-inspect reported exact layout php-2.11.8, a known bound queue, verified inline values, no unsupported state and no source mutation.
3. Source, byte-copy backup and separate restored copy all fingerprinted identically: sha256:9bccec26f129180ec76c24599379a731cbc1cd5349ffe66e17951f52e552568a (4 runs, 7 workflow tasks, 1 queued timer job).
4. Published 3.0.4 took over only the restored copy: status=taken_over, backend SQLite, exact source layout and fingerprint, backup_fingerprint_matched=true; the recorded worker incarnation was copied.
5. The same published image started from the converted copy and answered release readiness. The existing PHP-era work resumed:
   • activity: original schedule/start → completed → workflow completed; stale attempt refused 409;
   • timer: original timer fired once and the workflow completed;
   • signal: original receipt/wait advanced and applied once, then completed;
   • expired lease: redelivered as attempt 2, completed, and stale attempt refused 409.
6. Native stopped cleanly (exit 0, no OOM). PHP 2.11.8 refused the converted copy with php_database_refused. The untouched original still had the exact pre-backup fingerprint and PHP accepted it.
7. The acceptance result is outcome=pass (normalized receipt SHA-256 67bfcf0719699810144358e948621e6dcf536aba51bf95894df6d346fabd0f07). Every owned container, volume and network was removed; the unrelated existing Redis container was untouched.

One preliminary local run is excluded from acceptance: after successful takeover and native readiness, Docker Desktop did not expose a macOS /tmp receipt mount to the PHP resume helper. It produced no behavioral result and the cleanup trap removed every owned resource. Moving only that evidence directory under the shared worktree yielded the clean, exact rerun above; the product artifacts and procedure were unchanged.

Decision:

• This closes the missing published-artifact bridge when combined with the already-green exact-source backend matrix and the independently accepted amd64/arm64 published Compose smoke. It clears the installed-acceptance hold for the takeover command’s documented layouts, backends and refusal rules.
• It does not expand takeover beyond the current Status and scope table: external payload references and every other listed unqualified state remain refused; unknown layouts remain refused; the backup-first and stopped-PHP requirements remain mandatory.
• The current PR text is now stale where it says no published image has been accepted. Please update the exact head to cite this receipt and state the evidence precisely: installed 3.0.4 arm64/SQLite boundary here, plus the source CI matrix for the other listed backends and the separate published amd64/arm64 Compose smoke. Do not claim an installed all-backend matrix. The revised head needs a final bounded Security readback before merge.

[rmcdaniel]

Security re-review hold on exact head 52b0be1a2deb0b36a42756a6dba29f81854c5d20

The installed-acceptance description is otherwise scoped correctly, but the new exact-source CI citation is not valid evidence as written.

• Run 41597054274 does have head SHA 1f4ef0c7a745e1ebdc07cd38761cd3f666a36158, but Release 3.0.4: Published Compose default to 3.0.4 #991 changed only the Compose default. In that Server-parity run every native build, SQLite execution/cohort, PostgreSQL, MySQL and MariaDB job was skipped. Only the input/result shell jobs ran. It does not demonstrate the backend takeover matrix.
• The usable full-matrix evidence is merge-group run 41578731039, head 0d50047c1df4bb2adf905edaa779865abf080119, terminal success. Its SQLite execution/cohorts, two PostgreSQL cohorts, MySQL 8 cohorts and MariaDB 10.11 cohorts all ran and passed, alongside the full relevant matrix.
• That earlier head is valid for 3.0.4 only through exact content identity, which I independently read back: its rust tree is the same as 3.0.4 (9eff268fbeeb2e77aed9a484a7ebf3c2aaf3e56e), and the upgrade-source-profile.sh, author, resume helper and server-parity.yml blobs are byte-identical between 0d50047c and 1f4ef0c7.

Please remove run 41597054274 as matrix evidence and say precisely that run 41578731039 passed on the byte-identical 3.0.4 Rust/takeover-harness/workflow tree, naming the tree/blob binding above. Do not call the skipped #991 run a backend-matrix pass. This is a source-record blocker; the PR remains held until the corrected exact head and its checks return.

[rmcdaniel]

Security correction readback — exact head 8d927ddb0df4f8d8a5034089409c69e064eadb48

The matrix-evidence blocker is fixed.

• The skipped Release 3.0.4: Published Compose default to 3.0.4 #991 run is no longer presented as backend evidence.
• The page now cites full-matrix merge-group run 41578731039 at 0d50047c1df4bb2adf905edaa779865abf080119, whose SQLite, PostgreSQL, MySQL 8 and MariaDB 10.11 cohort jobs all ran and passed.
• I independently rechecked every content binding printed in the page: the 3.0.4 and matrix commits share Rust tree 9eff268fbeeb2e77aed9a484a7ebf3c2aaf3e56e, takeover-harness tree fa7ebe87acbb0eb381634098dcbae18b9a6b0545, and the four named profile/author/resume/workflow blobs. The commits differ only in the listed release and Compose workflow/script/default paths.
• The evidence separation remains precise: installed published 3.0.4 arm64/SQLite takeover; exact-content source matrix for the other listed backends/cohorts; dual-architecture published Compose startup, explicitly not takeover.
• The scope/refusal boundary remains unchanged. No installed all-backend claim, external-payload takeover claim or unknown-layout claim was introduced.

This exact head is source-clear. Its fresh hosted checks are still running; merge remains held until the same head has a terminal green set and receives final carry. Any push requires re-review.

[rmcdaniel]

Installed published-image takeover acceptance (native 3.0.4)

More installed-backend evidence, on top of Security's arm64/SQLite receipt (6600663947). This adds MySQL 8.0 and PostgreSQL 17 runs on the published images, plus a short supplementary SQLite run. Here the PHP-era state was written by a published SDK worker, and native was started with role tokens, a PHP-issued runtime credential and DW_SERVER_KEY. I followed the #992 runbook at 145ad1d6 as an operator would. The main output is the list of differences between the runbook text and what actually happened. Security has not reviewed this evidence, and it does not ask for any change to the scope.

Artifacts (all pinned by digest, linux/arm64, Colima on Apple Silicon)

Artifact	Index digest	arm64 child
native 3.0.4 (durableworkflow/server = ghcr.io/durable-workflow/server, revision 1f4ef0c7a745)	sha256:013010476b78f37038ac09d9498a95cdbf5ae7896111c321b2d4a2aac2f81f6b	sha256:c097590daee46a9ff0d4f5207e6a9a2b5dcb4826e30c8df60876b3be97b976d8
PHP Server 2.11.8	sha256:3848d5d5cd1b23bd2881e3fdecea64cc44b72e6fb812e577c2db6d0872d065dd	sha256:cea163f8fcedf3a17302acf0141b073a85fbdb395ebbcd59e5af6f5fa9ad75e1
mysql:8.0 (8.0.46)	sha256:7dcddc01f13bab2f15cde676d44d01f61fc9f99fe7785e86196dfc07d358ae2b	sha256:213bbfaf699693a40a20a12bb4342d2589a15a3dc7153db698eaed252a92458e
postgres:17 (17.11)	sha256:3b0cda93be7c8dba2ed484d19393d93845dc3dafb88973e248ba8e6d99d784e1	sha256:ac359420747e1f5f8ac4d23f40d7ce022c0a39090ed6cdd8dde5f596affbe41d
redis:7-alpine	sha256:8da19c7399dca900c6822a1cf30f9d2285585e2c4b457ff4095b0f4500054d3f	sha256:c1945f0eeb2d503f8421952da953a3804d38e4f004d4ef11d57826da59d2c332
SDK	Python SDK durable-workflow==2.13.0 from PyPI on python:3.12-slim@sha256:b18b261e1b4974b4f321451ad5251ded85abb22c2b8b262da527dda1d031ab30

Topology. PHP ran from the 2.11.8 docker-compose.published.yml, taken verbatim from tag 2.11.8: bootstrap, server, queue worker, scheduler, Redis queue and cache. A Compose override added only resource limits, init, labels, the image digests and loopback publishing. For PostgreSQL the override also swapped in a postgres:17 service with DB_CONNECTION=pgsql; for SQLite, DB_CONNECTION=sqlite and a named volume at /app/database. PHP served with random local values for DW_AUTH_TOKEN, DW_WORKER_TOKEN, DW_OPERATOR_TOKEN, DW_ADMIN_TOKEN and DW_SERVER_KEY, plus DW_RUNTIME_CREDENTIALS_ENABLED=true. None of these were printed.

Workers. Two long-lived Python SDK worker containers ran across the whole handoff, on the same container and process with no restart (RestartCount=0):

• tk queue, authenticating with DW_WORKER_TOKEN;
• tk-rc queue, authenticating with a runtime credential issued by PHP through PUT /api/runtime-credentials/{id} with the admin token.

The control client used DW_OPERATOR_TOKEN.

State created under PHP on every backend:

• 4 completed runs (3 on tk, 1 on tk-rc);
• a run waiting on a short timer (due during or just after the stop window) and one on a long timer;
• a run waiting on a signal (wait_condition);
• a run with an in-flight activity: claimed and still executing in the worker during the stop, takeover and start;
• a cron schedule (* * * * *, overlap skip, UTC) that had fired once under PHP.

On PostgreSQL I also added a second schedule and captured a PHP-issued schedule-list next_page_token.

Operator shell. Bash 5. Every runbook command was run verbatim. One wrapper function added --label, --init, --cpus/--memory/--pids-limit and --network <compose network> to each docker run and changed no runbook argument. Secrets were exported from a 0600 file instead of an interactive read -rs. The Environment block was used as written, including the empty-to-unset loop:

export IMAGE=durableworkflow/server@sha256:013010476b78f37038ac09d9498a95cdbf5ae7896111c321b2d4a2aac2f81f6b
export DB_CONNECTION=mysql DB_HOST=mysql DB_PORT=3306 DB_DATABASE=durable_workflow DB_USERNAME=workflow   # pgsql: DB_HOST=postgres DB_PORT=5432
export DB_PASSWORD=<redacted>
db="-e DB_CONNECTION -e DB_HOST -e DB_PORT -e DB_DATABASE -e DB_USERNAME -e DB_PASSWORD"
queues="-e DW_RUST_UPGRADE_QUEUE_CONNECTION=database -e DW_RUST_UPGRADE_QUEUE_DEFAULT_CONNECTION=database -e DW_RUST_UPGRADE_QUEUE_DEFAULT_QUEUE=default"
export DW_AUTH_TOKEN=<redacted> DW_WORKER_TOKEN=<redacted> DW_OPERATOR_TOKEN=<redacted> DW_ADMIN_TOKEN=<redacted>
export DW_RUNTIME_CREDENTIALS_ENABLED=true DW_SERVER_KEY=<redacted>
for v in DW_WORKER_TOKEN DW_OPERATOR_TOKEN DW_ADMIN_TOKEN DW_RUNTIME_CREDENTIALS_ENABLED DW_SERVER_KEY; do [ -n "${!v-unset}" ] || unset "$v"; done
serve="$db -e DW_AUTH_TOKEN -e DW_WORKER_TOKEN -e DW_OPERATOR_TOKEN -e DW_ADMIN_TOKEN -e DW_RUNTIME_CREDENTIALS_ENABLED -e DW_SERVER_KEY"

Results

Backend	Result
MySQL 8.0	PASS, including every post-marker statement I tested. There are documentation discrepancies; see D1–D4.
PostgreSQL 17	PASS
SQLite (supplementary to Security's receipt)	PASS

MySQL 8.0 (UTC, 2026-11-03)

1. Stop PHP, 10:05:28–10:05:46: docker compose stop server worker scheduler. The server exited 0. The queue worker and scheduler exited 137 (SIGKILL after Compose's 10 s grace). The processlist then showed no session on durable_workflow.

2. Inspect, 10:05:47: docker run --rm $db "$IMAGE" upgrade-inspect, the $queues upgrade-queue-inspect and upgrade-payload-inspect. All returned inspected with source_layout=php-2.11.8. queue_inspection.status=known_subset_bound, jobs=0, because the published Compose queues in Redis. Schedules 1 (0 unsupported), namespaces 1 (0 unsupported), no dispatches, no stream references, no pending deletions.

3. Back up and prove it, 10:06:02–10:06:18. The literal existing procedure cannot be used here; see D1. I used a plain mysqldump --single-transaction --quick --no-tablespaces through docker compose exec -T mysql as PHP's user. The footer was present, and stderr held only the password-on-CLI warning. I restored it into workflow_restored as root. docker run --rm $db -e DB_DATABASE=workflow_restored "$IMAGE" upgrade-fingerprint first failed with upgrade_fingerprint_unavailable (see D2). After GRANT SELECT, the source and the restored copy both gave sha256:3e2888cc6d4b806e2c096ea391a3da358b2e0af522387632c607afe7b2cd6d42 (27 ms).

4. Take over. Two refusals came first, both pre-marker, and both left the database unchanged:

   • At 10:06:30, the literal docker run --rm $db $queues -e DW_UPGRADE_BACKUP_FINGERPRINT "$IMAGE" upgrade-takeover with PHP's role was refused with upgrade_source_sessions_unverifiable: the published Compose user has no PROCESS (see D3). Afterwards dw_server_schema did not exist and the fingerprint was unchanged.
   • At 10:06:31, with an open SELECT SLEEP(25) session on durable_workflow and the root role, the takeover was refused with upgrade_source_in_use. Again there was no dw_server_schema and the fingerprint was unchanged.

   At 10:06:59, as root (DB_USERNAME/DB_PASSWORD overridden for this one command), the takeover exited 0 in under 1 s: status=taken_over, backend=mysql, source_layout=php-2.11.8, fingerprint=sha256:3e2888cc…, backup_fingerprint_matched=true, incarnations_copied=2. SELECT engine, version FROM dw_server_schema then returned rust-development 8.

5. Start native, 10:07:05, using the runbook command docker run -d --name durable-workflow-native $serve -v durable-workflow-native-storage:/app/storage -p 127.0.0.1:8080:8080 "$IMAGE" plus --network tk304-mysql_default --network-alias server (see D5). Native ran with PHP's own workflow role. /api/ready returned ready with qualification=release at 10:07:06.

6. Resume (10:07:25–10:07:28):

   • Workers: both existing SDK workers reconnected on their own after a ConnectError backoff, with no restart. The tk-rc worker, using the PHP-issued runtime credential, completed new work under native.
   • Short timer (scheduled under PHP and due at 10:07:06) fired under native. The run completed at 10:07:25 with echo:after-timer.
   • Signal: approve was sent under native, and the run completed with echo:approved-by:native-operator.
   • In-flight activity: the activity claimed under PHP at 10:04:37 kept running through stop, takeover and start (169 s). Its completion was accepted by native and the run completed.
   • New runs on tk and on tk-rc completed.
   • Schedule: it had fired at 10:05 under PHP. 10:06 fell in the stop window and was not backfilled (see O2). Native fired 10:07, 10:08 and 10:09 (fires_count 4).
   • History: a PHP-completed run read under native gives StartAccepted, WorkflowStarted, ActivityScheduled, ActivityStarted, ActivityCompleted, WorkflowCompleted.
   • Token roles under native match PHP's backward-compatible behaviour:
     • DW_AUTH_TOKEN: list workflows 200, runtime credentials 200;
     • operator token: 200 and 403;
     • admin token: 200 and 200;
     • worker token: 403 on the control plane.

7. PHP refuses, 10:09:36. After native stopped (exit 0, no OOM), docker compose run --rm --no-deps bootstrap (PHP 2.11.8) exited 1 with php_database_refused: This database has a Rust Server schema marker. PHP cannot use it….

MySQL/MariaDB post-marker behaviour. I tested this on restored copies, as Rehearsing on a restored copy describes. To widen the window, each copy carried 300k failed_jobs rows; the fingerprint then took about 1 s. I polled dw_server_schema from a session with no default database, and docker killed the takeover when the rust-takeover:php-2.11.8 row appeared at version 0, before the fenced check completed.

Check (UTC)	Runbook says	Observed
10:08:10, SELECT engine, version FROM dw_server_schema after the kill	marker row, version=steps done, 0 = fenced not passed	rust-takeover:php-2.11.8 0 ✅
10:08:27, native start on it	native_takeover_incomplete	exit 1, Error: Refused { status: 409, reason: "native_takeover_incomplete" } ✅ (see O3)
10:08:27, PHP 2.11.8 server:assert-php-database	PHP refuses from the marker on	exit 1, php_database_refused ✅
10:08:35, rerun with the same digest while another session is open	upgrade_source_in_use; stop the session and rerun	upgrade_source_in_use, marker still at 0 ✅
10:08:37, rerun with a different digest	upgrade_backup_fingerprint_mismatch; rerun with the original	upgrade_backup_fingerprint_mismatch, marker still at 0 ✅
10:08:50, rerun with the original digest	resumes and repeats the fenced check	exit 0, taken_over, converted.resumed_after_step=0, table_rows non-null (fenced not completed earlier, consistent with Success), marker rust-development 8 ✅
10:09:13–15, second copy: killed at version 0, one UPDATE written, rerun with the same digest	the fenced check refuses; only a restore works	upgrade_backup_fingerprint_mismatch, marker still rust-takeover:php-2.11.8 0 ✅
10:09:16, restore that copy's dump into a new database	the restore equals the digest; take over again from step 2	restored fingerprint equals the digest, and the takeover of the restore returned taken_over, backup_fingerprint_matched=true ✅

PostgreSQL 17 (UTC, 2026-11-03)

1. Stop PHP, 10:11:08–10:11:26: the same command and the same exit codes as on MySQL.

2. Inspect, 10:11:38: all three commands returned inspected with php-2.11.8 and known_subset_bound.

3. Back up, 10:11:39: pg_dump -Fc, then createdb workflow_restored and pg_restore --no-owner --exit-on-error. The backup doc has no PostgreSQL procedure; see D1. Source and restored copy both gave sha256:994370369fbfdd13405f1182252aff732bf085ca6a90e611c1744e1731a316f6 (21 ms).

4. Refusal path, 10:11:46. After the backup I wrote one UPDATE workflow_schedules SET note=… to the source. upgrade-takeover with the backup digest was refused with upgrade_backup_fingerprint_mismatch (exit 1). Afterwards to_regclass('public.dw_server_schema') IS NULL was true and the source fingerprint was unchanged (sha256:378fa8d5…), so the database was unchanged as stated.

   The runbook's safe action, "take a new backup and verify it again", worked: at 10:11:55 a new dump restored into workflow_restored2, and both fingerprinted sha256:378fa8d5861ed7a775c5565b4bf5a75a9f716b67e4a26cca41e4751f5fd5230f.

5. Take over, 10:11:56, with PHP's role, which is the owner: taken_over, backend=pgsql, backup_fingerprint_matched=true, incarnations_copied=2.

6. Start native, 10:11:56, with the same command as on MySQL. It was ready at 10:11:57.

7. Resume:

   • Short timer, due at 10:12:20, after native started: it fired under native and the run completed at 10:12:21.
   • Long timer (420 s, due 10:17:51): fired under native at 10:17:51, about 6 minutes after native started, and the run completed with echo:after-timer. This timer was neither overdue nor due at start.
   • Signal, in-flight activity (90 s held across the handoff), new runs on both workers (role token and PHP-issued runtime credential): all completed.
   • Schedule: it fired at 10:11 under PHP and at 10:12 and later under native.
   • History of a PHP-completed run was intact.
   • The PHP-issued schedule-list next_page_token was accepted by native and returned page 2 (tk-pg-sched). This is direct evidence that DW_SERVER_KEY carries over as the runbook says.

8. PHP refuses: after native stopped at 10:18 (exit 0, no OOM), docker compose run --rm --no-deps bootstrap exited 1 with php_database_refused. Both SDK workers had RestartCount=0, and the schedule's fires_count was 8 at 10:18:00.

SQLite (supplementary)

The run went from 10:00:28 to 10:03:09. The facts checked:

• File ownership after PHP: the file is 1000:1000 even though the Compose worker and scheduler run as root, and there were no -wal/-shm files after a clean stop.
• Refusal probes:
  • inspecting before upgrade-prepare-sqlite gave upgrade_source_sqlite_journal_unqualified;
  • a fingerprint from a writable mount gave upgrade_source_readonly_filesystem_required.
• Prepare: upgrade-prepare-sqlite returned previous_journal_mode=wal.
• Fingerprints: source and byte-copy backup both gave sha256:30852f43….
• Takeover: taken_over.
• Native start: native ran with the runbook's SQLite serve/$vol command.
• Resume:
  • the overdue timer fired;
  • the signal, the in-flight activity (236 s) and new runs on both the role-token and runtime-credential workers completed;
  • the schedule continued;
  • history was intact.
• PHP refusal: bootstrap was refused with php_database_refused.

My planned SQLite post-backup-write refusal did not execute: a quoting error in my harness meant the write never ran. That run was therefore the normal step-4 takeover against an unchanged source. The write-refusal is covered on PostgreSQL above and by Security's run.

Runbook discrepancies

D1. Step 3, "back up with the existing procedure", cannot be followed after step 1 and would break the fingerprint equality.

self-hosted-backup-and-restore.md is written for docker-compose.php.yml, which 2.11.8 installations do not have; they have docker-compose.published.yml. Its hold step is docker compose … exec -T server php artisan external-payloads:backup-hold acquire. After step 1 this fails with service "server" is not running.

To see what an operator who tried harder would get, I ran the hold commands with docker compose run --rm --no-deps server …. That starts a PHP role after step 1, and the acquire and release both write the source. The three fingerprints all differ:

• before the backup: a8002214…
• the source after the hold release: 3e2888cc…
• the dump restored under the active hold: d9620dbf…

So step 3's "the two fingerprints must be equal" cannot be met by following that procedure. A plain mysqldump without the hold gives equal fingerprints. That is safe here only because takeover refuses every external payload reference, so there are no objects to copy.

The backup doc also has no PostgreSQL procedure. I used pg_dump -Fc and pg_restore --no-owner, as upgrade-source-profile.sh does.

Suggest: give the exact dump and restore commands for MySQL and PostgreSQL in step 3, say to skip the backup hold with PHP stopped (or acquire and release it before fingerprinting, never across the dump), and use the Compose file name operators actually have.

D2. The restored database needs creating and granting, and the runbook doesn't say so.

Step 3's -e DB_DATABASE=workflow_restored … upgrade-fingerprint runs with PHP's role. On the published MySQL Compose, workflow has rights only on durable_workflow. Creating workflow_restored needs root, and fingerprinting it as workflow fails with the generic upgrade_fingerprint_unavailable until GRANT SELECT ON workflow_restored.*. PostgreSQL's POSTGRES_USER is a superuser, so this did not occur there.

D3. MySQL, step 4: the published Compose role lacks PROCESS.

Step 4 says to override DB_USERNAME/DB_PASSWORD "if PHP's role is not the owner". On the published MySQL Compose, workflow does own the tables, so an operator would not override, and the literal command is refused with upgrade_source_sessions_unverifiable. The refusal is safe and matches the table. Still, step 4 should say plainly that the published MySQL Compose needs the root account (or GRANT PROCESS) for the takeover.

It should also show how to override the password without putting it on argv. For example: DB_USERNAME=root DB_PASSWORD="$ROOT_PW" docker run … -e DB_USERNAME -e DB_PASSWORD …, with the root password read by read -rs.

D4. The queue variables don't fit the published topology.

The queues block hardcodes DW_RUST_UPGRADE_QUEUE_DEFAULT_CONNECTION=database, while the table says it is PHP's QUEUE_CONNECTION. For the published 2.11.8 Compose that is redis. With Redis, the jobs table is empty and both values are accepted (known_subset_bound, jobs=0). Redis-held wakeups are not carried, yet timers still fired from the durable rows under native on all three backends.

The runbook should cover the Redis case: what to set, and that Redis contents are not migrated and not needed. It should not show database as a value to copy.

D5. The start command doesn't say how existing workers reach native.

Workers and clients address the PHP server service, or whatever proxy or host port they used. Native has to answer at that same address. With the published Compose I had to add --network <compose network> --network-alias server. The runbook mentions --network only for reaching the database.

Suggest: one sentence telling operators to start native at the address workers and clients used for PHP (the same alias, proxy upstream or host port), with PHP's server stopped so the port is free.

D6. Stale version and status text.

• "The command ships in the native 3.0.0, 3.0.1 and 3.0.2 images." 3.0.4 is published, and it is the image accepted here.
• IMAGE=…@sha256:<index digest from the release notes>: tag 3.0.4 exists, but there is no GitHub release 3.0.4 yet (the latest release is 3.0.2). So the digest cannot come from release notes; I resolved it from both registries, and they are identical.
• The "Not yet qualified on a published image" call-out and the What is not qualified yet bullets are stale since Security's receipt, as Security already noted.

D7. Minor precision items.

• Session check: on MySQL, upgrade_source_in_use counts only sessions whose current database is the target (PROCESSLIST.DB = DATABASE()). A session with no default database, using schema-qualified SQL, is not counted. My polling session was not counted, as expected. The fingerprint still catches pre-marker writes. Say "connected with this database selected".
• Process stop: after docker compose stop, the PHP queue worker and scheduler exit 137, not 0. That is harmless here: the fingerprint was stable and the backup equal. The "Stop PHP" step could say that 137 is expected for those two roles, or give a longer -t.

Observations, not runbook errors:

• O1. Run IDs created by native are upper-case ULIDs (01M6GE1G…); PHP's are lower-case (01m6gdw7…). Clients that compare IDs case-sensitively will see mixed casing after takeover.
• O2. Cron fires that fall due during the stop window are not backfilled under native. On MySQL, 10:05 fired under PHP, 10:06 was missed and 10:07 fired under native. Worth one line under Schedules.
• O3. Native's startup refusal prints Error: Refused { status: 409, reason: "…" }, not a bare reason line.

Verified as written:

• the empty-to-unset loop and the $serve environment (DW_AUTH_TOKEN backward compatibility, role tokens, runtime credentials, DW_SERVER_KEY page tokens);
• SQLite ownership (no chown needed) and the :ro/WAL rules;
• on SQLite and PostgreSQL, a refusal leaves the database unchanged;
• on MySQL, a refusal before the marker leaves it unchanged, and every post-marker statement in Refusals that I tested matched (see the MySQL post-marker table);
• the success report fields, including resumed_after_step and table_rows;
• PHP's refusal after takeover.

Cleanup

Every container, network and volume I created carried the label com.durable-workflow.task=general-takeover-304. That includes the Compose projects tk304-{sqlite,mysql,pgsql}, the SDK workers, durable-workflow-native, the storage and backup volumes, and the anonymous native volumes. All were removed after each backend. After the last teardown at 10:18 UTC, a label filter lists 0 containers, 0 volumes and 0 networks, and nothing is left over against the pre-run baseline. The locally built SDK image was removed too. The unrelated pre-existing wf-agent-redis container was untouched. Raw logs are kept privately on lab2; tokens and keys were never printed or posted.

🤖 Generated with Claude Code

[rmcdaniel]

Security re-review of exact head 5be81513df70837579c7485c23e1838d9c63555c: hold.

I independently read the full installed MySQL 8.0/PostgreSQL 17 receipt in 6600980478. Its immutable PHP/native/database tuple matches the published artifacts already accepted for 3.0.4, its takeover/refusal outcomes are internally consistent, and its limits are explicit. I accept that receipt as bounded installed evidence for PHP 2.11.8 → native 3.0.4 on those two arm64 backends. That means the new text saying Security has not reviewed those runs will itself be stale on the correction head; change it to bounded Security acceptance, retaining the MariaDB/amd64/other-PHP exclusions.

The operational corrections are directionally right, but four issues remain:

1. Blocker — the MySQL commands corrupt the credentials and arguments. Inside the single-quoted sh -c strings, the new " characters are literal input to the inner shell. For example, MYSQL_PWD="$MYSQL_PASSWORD" assigns a value that includes quote characters; -u"$MYSQL_USER" also splits a username containing whitespace rather than quoting it. I reproduced the parse locally without printing any credential. Remove the backslashes inside the single quotes:

   • sh -c 'MYSQL_PWD="$MYSQL_PASSWORD" exec mysqldump -u"$MYSQL_USER" ... "$MYSQL_DATABASE"'
   • sh -c 'MYSQL_PWD="$MYSQL_ROOT_PASSWORD" exec mysql -uroot'
   • and the restore form likewise.
     In the Markdown source those should contain ordinary inner double quotes, not backslash-double-quote sequences.

2. Blocker — net=<compose project>_default is not an assignment an operator can execute. The shell parses <compose as input redirection. Use a syntactically valid example or, preferably, an exact read-only resolution from the running database service followed by docker network inspect "$net". Do not let a multi-network result silently become an unquoted $db argument list.

3. Blocker — the Design section still contradicts the fixed procedure. Its step 3 still says to take the backup with the “existing procedure.” That is exactly D1's unsafe/unworkable path: after PHP is stopped the hold cannot be acquired, and acquiring/releasing it via a one-off PHP role writes the source and breaks fingerprint equality. Make Design step 3 point to this page's stopped-PHP engine-specific dump/restore path and state that no external-payload hold is used because takeover refuses every external reference.

4. Scope correction — narrow the Redis claim. The installed runs observed an empty database jobs table and proved that the tested timer/activity/workflow cohort resumed from canonical SQL rows; they did not inventory arbitrary Redis contents. Say that PHP Server's Redis wakeups are not migrated and are not required by native for the qualified cohort/canonical durable tasks. Preserve Redis for PHP rollback and any unrelated application jobs. Avoid the absolute “jobs held in Redis ... are not needed.”

Also make the stop text observational rather than universal: the published Compose worker/scheduler may exit 137 after the default grace (and did in the installed runs); an operator must still confirm every writer is gone. The rest of D1–D7 is represented accurately, including PROCESS/root, restored-database access, worker address continuity, session scope, schedule non-backfill, release identity and refusal output.

Nothing may merge from this head. Please send the next exact head; I will re-run the command parsing and contradiction checks before source clearance.

[rmcdaniel]

Security source re-review of exact head ea42adc386bccb391a53ca2020e06dfba8fde16a: source clear; final hosted carry still pending.

The 5be81513 hold is resolved:

• the MySQL sh -c bodies now contain ordinary inner double-quote bytes, not backslash-quote bytes. A harmless local parse control with spaces in the synthetic password, username and database kept the password value exact and kept each username/database argument whole;
• the Compose network is resolved read-only from the stopped server service, rejects empty/multi/unsafe output, and is checked with docker network inspect before it enters $db;
• the Design section now uses the same stopped-PHP engine dump/restore procedure and explicitly refuses the general external-payload hold for this takeover boundary;
• Redis language is limited to PHP Server wakeups and the qualified canonical-row cohort, discloses that Redis contents were not inventoried, and preserves Redis data for PHP rollback/unrelated jobs;
• exit 137 is recorded as the installed observation/possibility, followed by an explicit all-writers-gone check;
• the accepted MySQL 8.0/PostgreSQL 17 evidence is now described as bounded PHP 2.11.8 → native 3.0.4 arm64 evidence, while MariaDB, amd64 and other PHP versions remain excluded.

Independent checks:

• git diff --check is clean;
• every sh fence passes bash -n after substituting only the runbook's explicit operator placeholders;
• a synthetic merge of this head into current protected main fcb25574766e4f936b330dcd6b62a62f34c45eec is conflict-free and changes exactly docs/rust-server-storage-audit.md, docs/rust-server-upgrade-takeover.md, and docs/rust-server-upgrade.md;
• no runtime, workflow, schema, Compose, credential or release artifact changes are introduced by that merged tree.

This is bounded source clearance, not authorization to merge yet. Fresh hosted checks on ea42adc3 are still running. Keep it out of the queue until Security records the final exact-head carry with zero failing/pending checks.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[rmcdaniel]

Security: please review exact head ea42adc386bccb391a53ca2020e06dfba8fde16a. It replaces the held 5be81513, which itself moved off the cleared 8d927ddb to fix the D1–D7 findings from the installed MySQL and PostgreSQL runs. All five points in your hold 6601096208 are addressed; the PR body's Update at ea42adc3 section lists each fix.

• The MySQL sh -c commands are one line each. I ran them verbatim in a disposable Compose project with a dummy password containing spaces, both quote kinds, $ and a backslash. Every value arrived exactly.
• net is resolved from the server service, validated to a single name, and checked with docker network inspect.
• Design step 3 no longer points to the backup hold.
• The Redis claim is narrowed to what the runs observed.
• The exit-137 note is now an observation.

Your MySQL/PostgreSQL acceptance is cited as bounded arm64 evidence for PHP 2.11.8 to native 3.0.4. MariaDB, amd64 and other PHP releases are still excluded. CI is green on this head. It is not queued and auto-merge is off.

[rmcdaniel]

Security final carry — exact head ea42adc386bccb391a53ca2020e06dfba8fde16a.

The source clearance remains recorded here. I have now independently read back the exact head and the full hosted check rollup: 15 success, 8 intentionally skipped, 0 failed, 0 cancelled, 0 pending. The last two gates were the PHPUnit feature suite (15m08s, success) and Feature source qualification (success).

The synthetic merge into protected main fcb25574766e4f936b330dcd6b62a62f34c45eec remains conflict-free and changes exactly the three reviewed documentation paths, with no runtime, workflow, schema, Compose, credential, or release-artifact change.

Final Security clearance: General may queue this exact head only. Any head change, merge conflict resolution, or expanded diff requires a fresh readback.

[rmcdaniel]

Security verified merged outcome: protected main is now signed merge commit a051f57fcc7e3605b301581ee9105461b3ef5815, with parents fcb25574766e4f936b330dcd6b62a62f34c45eec and exact cleared head ea42adc386bccb391a53ca2020e06dfba8fde16a. Its tree is fcfbd18f401b893644a43b85912a52c124ba946e, exactly the synthetic merge tree recorded during clearance. GitHub reports only the three reviewed documentation paths changed. The 15-success/8-intentional-skip check rollup is unchanged. This closes Security carry for #992.