Skip to content

Qualify installed command response-loss recovery - #320

Merged
rmcdaniel merged 7 commits into
mainfrom
sdk-python/216-installed-response-loss
Nov 13, 2026
Merged

rmcdaniel merged 7 commits into
mainfrom
sdk-python/216-installed-response-loss

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Qualify recovery of one retained signal/update admission after the successful upstream response is lost. The published Python Client and actual Worker use the accepted shared loopback fixture. No product code or capability changes.

The correction for Security's provenance HOLD captures owner/no-follow/single-link inputs against an exact Git commit before runtime start, executes/imports/reads only captured bytes, and rechecks path/inode/content identities after return. A fixed published Python image replaces arbitrary host interpreters; a fresh all-wheel/hash-locked registry environment is fingerprinted around execution. The controller uses an empty-proxy readiness opener and exact two-container/token cleanup.

The six-cell/eight-loss observation atd4dbdaf is independently coherent but remains qualification-HOLD. The first corrected bootstrap stopped before admission on noexec scratch and remains NONPASS; its cleanup/rechecks passed. The successor keeps noexec and closes fresh virtualenv links to the immutable image interpreter. All61 inert controls and Ruff pass, including substitution/rename/symlink cases and the predecessor's forged-report boundary. Corrected exact-source receipt atc1e8a09 repeats all8 loss outcomes with7 captured input roles, fixed published Python3.12.16, a fresh hash-locked registry installation, equal1164-file inventories and verified cleanup. Independent Security counter-review remains required.

Next owner/action: sdk-python repeats the corrected exact source on registry2.17.2/qualified native3.9.2, sends the bound receipt for independent Security counter-review, fixes findings and merges only with exact-head required checks green. Legacy physical/synthetic and namespace/run cells remain separate under shared216; SDK321 is held on its recorded producer/parent gates. No full216 PASS or new package release is claimed.

Tracks durable-workflow/.github#216.

Latest frozen head3b207a0 closes the remaining name-cleanup finding with full creation IDs and ownership labels. Fresh ID-bound receipt repeats all8 loss outcomes with recorded ID-only cleanup. All72 inert controls and all20 exact-head checks pass. Independent counter-review remains the final source/qualification gate; earlier c1e8a09 is preserved under its specific cleanup HOLD.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Initial installed attempt at eebb5c9, 2026-11-13 16:08:41–16:08:43 UTC: RUNNER-BLOCKED / NONPASS, before any command admission. The read-only container root had no writable SQLite database path, so the first start returned503 storage_unavailable. This does not establish a product failure. The runner now declares bounded writable database/storage/tmp paths under its existing memory cap.

The cleanup verifier also expected different capitalization/wording for Docker29 missing-object replies and initially reported incomplete cleanup. Independent exact-name inspection plus the owned container's destroy event and its two associated volume identities confirm container and both volumes absent; generated credential absent, no OOM. The original incomplete receipt remains unchanged and a separate correction is retained. No PASS is inferred. Regression controls now cover genuine missing-object versus daemon-error replies and image tuple mismatches. sdk-python owns a fresh corrected-source repeat on the unchanged published package/image/proxy tuple.

@rmcdaniel

rmcdaniel commented Nov 13, 2026 •

Copy link
Copy Markdown
Member Author

Qualification HOLD after Security provenance review. The six-cell observation below independently reconciles, but hash-then-reopen execution must be corrected and repeated before runner/installed-cohort qualification. This is not a product failure. The original runner-blocked attempt remains separate.

Installed Python response-loss cohort PASS, frozen public runner d4dbdaf259f6041f10bc96fbb6672d52a9ef4d94 / tree cc1026f458f46da9158f62cab89cb74642ef5f2e, draft SDK320. Ran 2026-11-13T16:13:02.975686+00:00–2026-11-13T16:13:06.823779+00:00, Linux amd64/SQLite, against the independently qualified immutable native3.9.2 tuple below. This is published registry Python2.17.2, not a source overlay. All46 installed SDK files equal the approved PyPI wheel, and dependency/interpreter identities are recorded.

Six workflows cover eight complete upstream successes intentionally lost before the client receives a status line. Automatic retries have exactly two byte-identical sends with one caller key; manual max-attempts-one failures expose that key for explicit recovery. Every command survives four concurrent identical retries, refuses changed intent as a typed conflict, admits/executes once, and recovers the same identities after closure or continuation with zero extra historical admissions. Explicit keys are 255 Unicode code points /687 UTF-8 bytes for both routes, confirming the corrected producer boundary for this installed Python tuple. Nested bytes, booleans, integers and Unicode preserve their types.

The accepted proxy is central117118c83093822469b3824ba9587ff4068ed29f /SHA256 bd1e8ccbbf76f63fc2eafa278a023eb181d405b68d9c30016392e395a1eff725. Independent sender/proxy request counts, lengths and hashes agree. Evidence retains only synthetic identities and bounded metadata, no credentials/headers/payload bodies. The actual Worker uses the registry package. Both consumer/controller complete flags are true; shutdown, noOOM and exact runtime/token cleanup are verified. The final controller uses bounded tmpfs paths, so no anonymous volumes remain. Twenty-four inert identity/refusal controls and Ruff pass.

The initial runner-blocked attempt remains NONPASS with its independent cleanup correction. Exact-head CI and independent source/evidence review still gate runner qualification/merge. sdk-python owns those fixes and the remaining legacy/readback acceptance using the accepted common helper. Server1210 retains real physical3.8 upgrade ownership; namespace/run fence and synthetic ambiguity/legacy cells remain separately scoped. This is not full216 closure. Private diagnostic retention is at most seven days under sdk-python ownership.

{
  "source_commit": "d4dbdaf259f6041f10bc96fbb6672d52a9ef4d94",
  "source_tree": "cc1026f458f46da9158f62cab89cb74642ef5f2e",
  "file_sha256": {
    "scripts/conformance/command_keys/response_loss.py": "f1a3c822412c033f258e8c60e6d8d792f0144d375e60ed57b57e35ed6c4f6847",
    "scripts/conformance/command_keys/run_response_loss.py": "d7370f7f2f38d1493f0b3b96dc2e67abe97637bfde9f240ee95fdd2554b7a948",
    "tests/test_command_keys_acceptance.py": "cd3637769d84adfec97e53020f80f6d957dcf145125fcaadaeaae50dfa4d29ab"
  },
  "package": {
    "version": "2.17.2",
    "wheel_sha256": "7dc77d3882fa5cbddbeaee6a4cc63b05285e280523c8ea40dc31e03cc12a40bc",
    "runtime_files_verified": 46,
    "python": "3.13.18",
    "dependencies": {
      "httpx": "0.28.1",
      "fastavro": "1.13.1"
    }
  },
  "expected_runtime": {
    "version": "3.9.2",
    "source": "5614274d2629d03f92932a07dfee0fb10349ddd9",
    "index": "sha256:8ffcdb2a36af3129a1987849aafe8715fe0d3bf9a9648467f0b5cb8eeef43ba9",
    "linux_amd64": "sha256:813c9337c44be7d497efd29e1b463546519e488aa796b5ecf9dfc6f4f10d2128",
    "qualification": "https://github.com/durable-workflow/server/pull/1209#issuecomment-6858140177"
  },
  "observed_image": {
    "id": "sha256:8ffcdb2a36af3129a1987849aafe8715fe0d3bf9a9648467f0b5cb8eeef43ba9",
    "architecture": "amd64",
    "os": "linux",
    "revision": "5614274d2629d03f92932a07dfee0fb10349ddd9",
    "repo_digests": [
      "ghcr.io/durable-workflow/server@sha256:8ffcdb2a36af3129a1987849aafe8715fe0d3bf9a9648467f0b5cb8eeef43ba9"
    ]
  },
  "started_at": "2026-11-13T16:13:02.975686+00:00",
  "finished_at": "2026-11-13T16:13:06.823779+00:00",
  "commands": [
    {
      "cell": 1,
      "command": "signal",
      "automatic": true,
      "explicit": false,
      "continued": false,
      "key_characters": 50,
      "key_utf8_bytes": 50,
      "proxy_attempts": 2,
      "identical_concurrent_retries": 4,
      "changed_intent": "request_id_conflict",
      "transport_exception": null,
      "identity": {
        "command_id": "01M7ATY4W7AB33XTZKYGWMQ8MW",
        "command_sequence": 2,
        "run_id": "01M7ATY4PXGR6GXD69Y5MH0SQY",
        "resolved_run_id": "01M7ATY4PXGR6GXD69Y5MH0SQY"
      },
      "original_response": {
        "upstream_status": 202,
        "upstream_complete": true,
        "upstream_body_bytes": 2180,
        "upstream_body_sha256": "da73216552e9a238f7c324ba2a23f908bc905496d3e144bbdd6cca3839ff4eb8",
        "dropped": true
      },
      "request_body_sha256": "ecf083ddc26414ade425eec1173f3ffd546ad76fc85dc559844ab32a709ee048",
      "request_body_bytes": 254
    },
    {
      "cell": 1,
      "command": "update",
      "automatic": true,
      "explicit": false,
      "continued": false,
      "key_characters": 50,
      "key_utf8_bytes": 50,
      "proxy_attempts": 2,
      "identical_concurrent_retries": 4,
      "changed_intent": "request_id_conflict",
      "transport_exception": null,
      "identity": {
        "command_id": "01M7ATY51TVWEJPRZ3X4D4JARW",
        "command_sequence": 3,
        "run_id": "01M7ATY4PXGR6GXD69Y5MH0SQY",
        "resolved_run_id": "01M7ATY4PXGR6GXD69Y5MH0SQY",
        "update_id": "01M7ATY51TVWEJPRZ3X4D4JARX",
        "accepted_at": "2026-11-13T16:13:03.418Z"
      },
      "original_response": {
        "upstream_status": 202,
        "upstream_complete": true,
        "upstream_body_bytes": 2896,
        "upstream_body_sha256": "413fcb28384a3a99d2d9945f949dfed00b5e4f76b4a7f9bc7756a0fd36d01b88",
        "dropped": true
      },
      "request_body_sha256": "8dc88397552d54c071f171ce759808a607b3320468faaed444632c163b8e1a84",
      "request_body_bytes": 276
    },
    {
      "cell": 2,
      "command": "signal",
      "automatic": true,
      "explicit": true,
      "continued": true,
      "key_characters": 255,
      "key_utf8_bytes": 687,
      "proxy_attempts": 2,
      "identical_concurrent_retries": 4,
      "changed_intent": "request_id_conflict",
      "transport_exception": null,
      "identity": {
        "command_id": "01M7ATY5M4JGBDGVM5WA648ZYK",
        "command_sequence": 2,
        "run_id": "01M7ATY5EWK5JNE9378614CXR6",
        "resolved_run_id": "01M7ATY5EWK5JNE9378614CXR6"
      },
      "original_response": {
        "upstream_status": 202,
        "upstream_complete": true,
        "upstream_body_bytes": 2817,
        "upstream_body_sha256": "5ce30436dd8443db23726dbffb0d6f573cffe71f163a01681f22772a70c68391",
        "dropped": true
      },
      "request_body_sha256": "ec5f4ab64d78f815e386557a2136b9caef2a1350d6371e01626bf4ec1fdd30bc",
      "request_body_bytes": 891
    },
    {
      "cell": 2,
      "command": "update",
      "automatic": true,
      "explicit": true,
      "continued": true,
      "key_characters": 255,
      "key_utf8_bytes": 687,
      "proxy_attempts": 2,
      "identical_concurrent_retries": 4,
      "changed_intent": "request_id_conflict",
      "transport_exception": null,
      "identity": {
        "command_id": "01M7ATY5SZ7F1YRA80E3YGPNVS",
        "command_sequence": 3,
        "run_id": "01M7ATY5EWK5JNE9378614CXR6",
        "resolved_run_id": "01M7ATY5EWK5JNE9378614CXR6",
        "update_id": "01M7ATY5SZ7F1YRA80E3YGPNVT",
        "accepted_at": "2026-11-13T16:13:04.191Z"
      },
      "original_response": {
        "upstream_status": 202,
        "upstream_complete": true,
        "upstream_body_bytes": 3533,
        "upstream_body_sha256": "9861778a47b5d910a92e4e78f57a9fc59d93ed4ec6c6b831f3c2e7c660771ed6",
        "dropped": true
      },
      "request_body_sha256": "d63c0290d9734469443a467feaa745182765b1a43ef74d660c0d2f481dc05404",
      "request_body_bytes": 913
    },
    {
      "cell": 3,
      "command": "signal",
      "automatic": false,
      "explicit": false,
      "continued": false,
      "key_characters": 50,
      "key_utf8_bytes": 50,
      "proxy_attempts": 1,
      "identical_concurrent_retries": 4,
      "changed_intent": "request_id_conflict",
      "transport_exception": "RemoteProtocolError",
      "identity": {
        "command_id": "01M7ATY6NSJXKTGV4XZC1G7KE7",
        "command_sequence": 2,
        "run_id": "01M7ATY6GV80ASQ649B506A70B",
        "resolved_run_id": "01M7ATY6GV80ASQ649B506A70B"
      },
      "original_response": {
        "upstream_status": 202,
        "upstream_complete": true,
        "upstream_body_bytes": 2180,
        "upstream_body_sha256": "d86e9b4a17d5a2723af21005d4441811e9de3c860a86a37c9c898256fa906703",
        "dropped": true
      },
      "request_body_sha256": "4634d311a95f4eb0749e3f7638b934ccd1039184c3082a584d222977f30a36cc",
      "request_body_bytes": 254
    },
    {
      "cell": 4,
      "command": "signal",
      "automatic": false,
      "explicit": true,
      "continued": false,
      "key_characters": 255,
      "key_utf8_bytes": 687,
      "proxy_attempts": 1,
      "identical_concurrent_retries": 4,
      "changed_intent": "request_id_conflict",
      "transport_exception": "RemoteProtocolError",
      "identity": {
        "command_id": "01M7ATY73T6E0CSC1V769P0V1K",
        "command_sequence": 2,
        "run_id": "01M7ATY6YSJ0MEJ9JDW4R544GA",
        "resolved_run_id": "01M7ATY6YSJ0MEJ9JDW4R544GA"
      },
      "original_response": {
        "upstream_status": 202,
        "upstream_complete": true,
        "upstream_body_bytes": 2817,
        "upstream_body_sha256": "263f09bf6b7f19cd09097c77ef4fe595317bb12e15e6f5e654d8834e985e20bc",
        "dropped": true
      },
      "request_body_sha256": "f5a03fbe45ff61ca88c3e2e7dc4872a4d04199beeee800c58e810938d59492f2",
      "request_body_bytes": 891
    },
    {
      "cell": 5,
      "command": "update",
      "automatic": false,
      "explicit": false,
      "continued": false,
      "key_characters": 50,
      "key_utf8_bytes": 50,
      "proxy_attempts": 1,
      "identical_concurrent_retries": 4,
      "changed_intent": "request_id_conflict",
      "transport_exception": "RemoteProtocolError",
      "identity": {
        "command_id": "01M7ATY7K0HHVJCZNFZ4P3GYX1",
        "command_sequence": 2,
        "run_id": "01M7ATY7DRWKJJBN2T6EQSSDNM",
        "resolved_run_id": "01M7ATY7DRWKJJBN2T6EQSSDNM",
        "update_id": "01M7ATY7K0HHVJCZNFZ4P3GYX2",
        "accepted_at": "2026-11-13T16:13:06.016Z"
      },
      "original_response": {
        "upstream_status": 202,
        "upstream_complete": true,
        "upstream_body_bytes": 2896,
        "upstream_body_sha256": "5add82110a31228716c6bc5f2d87b646ad386f646b07adac704f13588d877672",
        "dropped": true
      },
      "request_body_sha256": "7f129b1ed86132e032184354a77ead1d465f616d17e6f158c7ffde89f4f54cea",
      "request_body_bytes": 276
    },
    {
      "cell": 6,
      "command": "update",
      "automatic": false,
      "explicit": true,
      "continued": false,
      "key_characters": 255,
      "key_utf8_bytes": 687,
      "proxy_attempts": 1,
      "identical_concurrent_retries": 4,
      "changed_intent": "request_id_conflict",
      "transport_exception": "RemoteProtocolError",
      "identity": {
        "command_id": "01M7ATY823A172BNR3F4DQ7VMS",
        "command_sequence": 2,
        "run_id": "01M7ATY7X3W4D0XT8HZ6TX9YM2",
        "resolved_run_id": "01M7ATY7X3W4D0XT8HZ6TX9YM2",
        "update_id": "01M7ATY823A172BNR3F4DQ7VMT",
        "accepted_at": "2026-11-13T16:13:06.499Z"
      },
      "original_response": {
        "upstream_status": 202,
        "upstream_complete": true,
        "upstream_body_bytes": 3533,
        "upstream_body_sha256": "58e616675653f3eee89b650c7d4e0de728d1a1b67ee55a2a940c3b557927f53e",
        "dropped": true
      },
      "request_body_sha256": "1363eefab5280310d77bd8fb4a2f717c782d46ce18309a47c8ed7b6c0edf9378",
      "request_body_bytes": 913
    }
  ],
  "event_counts": [
    {
      "StartAccepted": 1,
      "WorkflowStarted": 1,
      "ConditionWaitOpened": 2,
      "TimerScheduled": 2,
      "SignalReceived": 2,
      "MessageCursorAdvanced": 3,
      "ConditionWaitSatisfied": 2,
      "TimerCancelled": 2,
      "UpdateAccepted": 1,
      "UpdateApplied": 1,
      "UpdateCompleted": 1,
      "WorkflowCompleted": 1
    },
    {
      "StartAccepted": 1,
      "WorkflowStarted": 1,
      "ConditionWaitOpened": 2,
      "TimerScheduled": 2,
      "SignalReceived": 2,
      "MessageCursorAdvanced": 3,
      "SignalApplied": 2,
      "ConditionWaitSatisfied": 2,
      "TimerCancelled": 2,
      "UpdateAccepted": 1,
      "UpdateApplied": 1,
      "UpdateCompleted": 1,
      "WorkflowContinuedAsNew": 1
    },
    {
      "StartAccepted": 1,
      "WorkflowStarted": 1,
      "ConditionWaitOpened": 2,
      "TimerScheduled": 2,
      "SignalReceived": 2,
      "MessageCursorAdvanced": 2,
      "SignalApplied": 2,
      "ConditionWaitSatisfied": 2,
      "TimerCancelled": 2,
      "WorkflowCompleted": 1
    },
    {
      "StartAccepted": 1,
      "WorkflowStarted": 1,
      "ConditionWaitOpened": 2,
      "TimerScheduled": 2,
      "SignalReceived": 2,
      "MessageCursorAdvanced": 2,
      "SignalApplied": 2,
      "ConditionWaitSatisfied": 2,
      "TimerCancelled": 2,
      "WorkflowCompleted": 1
    },
    {
      "StartAccepted": 1,
      "WorkflowStarted": 1,
      "ConditionWaitOpened": 1,
      "TimerScheduled": 1,
      "UpdateAccepted": 1,
      "UpdateApplied": 1,
      "UpdateCompleted": 1,
      "MessageCursorAdvanced": 2,
      "SignalReceived": 1,
      "SignalApplied": 1,
      "ConditionWaitSatisfied": 1,
      "TimerCancelled": 1,
      "WorkflowCompleted": 1
    },
    {
      "StartAccepted": 1,
      "WorkflowStarted": 1,
      "ConditionWaitOpened": 1,
      "TimerScheduled": 1,
      "UpdateAccepted": 1,
      "UpdateApplied": 1,
      "UpdateCompleted": 1,
      "MessageCursorAdvanced": 2,
      "SignalReceived": 1,
      "SignalApplied": 1,
      "ConditionWaitSatisfied": 1,
      "TimerCancelled": 1,
      "WorkflowCompleted": 1
    }
  ],
  "proxy_thread_stopped": true,
  "cleanup": {
    "container_absent": true,
    "volumes_removed": 0,
    "volumes_remaining": 0,
    "credential_removed": true
  },
  "oom_killed": false
}

@rmcdaniel
rmcdaniel marked this pull request as ready for review November 13, 2026 16:20
@rmcdaniel

Copy link
Copy Markdown
Member Author

Security exact-head review of d4dbdaf259f6041f10bc96fbb6672d52a9ef4d94 / tree cc1026f458f46da9158f62cab89cb74642ef5f2e: HOLD on the public runner source and its installed-acceptance disposition.

The retained run itself is coherent. I independently re-read run.json, consumer.json, the private log and the installed environment: six cells/eight deliberately lost complete 2xx responses, the published wheel SHA 7dc77d38…, 46 matching installed SDK files, accepted proxy SHA bd1e8ccb…, qualified native3.9.2 index 8ffcdb2a…, no OOM and exact runtime/token cleanup all match the public receipt. The initial storage attempt remains properly recorded as NONPASS. This finding is not a product failure.

One trust boundary remains blocking: the runner hashes reviewed inputs and then executes or imports them through separately resolved paths.

  • run_response_loss.py:75-82 hashes response_loss.py, but :134-152 later invokes that pathname with an arbitrary --python pathname. Neither the interpreter nor adapter is opened no-follow/owner/single-link, held, copied from captured bytes into the private run root, or identity-rechecked around the call.
  • The same controller passes args.proxy.resolve() and args.wheel.resolve() without holding either input. response_loss.py:53-66 hashes the wheel and then reopens the path with ZipFile; :216-222 hashes the proxy and then imports the path separately. A replacement between those operations can make executed proxy code or inspected wheel bytes differ from the reviewed bytes.
  • The controller then trusts consumer.json after checking only complete and the adapter-reported hash (run_response_loss.py:162-164). A substituted adapter is the party producing both fields, so that comparison does not close the execution gap. The current tests cover installed-file content substitution and image/cleanup refusals, but no adapter/proxy/wheel/interpreter path substitution.

Please make the exact executed inputs authoritative before the runtime starts: open no-follow from owner-controlled parents, require regular single-link owner/mode identities, capture and hash the adapter/proxy/wheel, and execute/import/read only a private immutable copy or held identity derived from those captured bytes. Bind the registry interpreter/venv provenance rather than accepting an unexamined executable path, and recheck held identities after the consumer. Add predecessor-failing symlink/replacement/rename controls for each executable input and prove that no substituted adapter can self-certify consumer.json. An explicit empty-proxy opener for the controller's loopback readiness check would also make its minimal-environment claim true in-process.

Preserve the current successful observation and the earlier NONPASS separately. A corrected exact-source repeat is required before this reusable runner or installed cohort can receive PASS. This does not affect the independently published Python2.17.2 package or Server3.9.2 qualification, and it is not full shared #216 disposition.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Provenance correction is pushed at exact 089fe4689f3803b27d2f6ce84a9f855e8d416ed1. The host interpreter/venv option is removed. A fixed published official Python3.12.16 image and fresh wheel-only, all-dependency hash-locked registry install replace it. Before runtime start, the controller loads its guard from the exact Git object, requires every executable source file to match that declared commit, and captures no-follow owner/single-link files with held descriptors. Adapter/proxy code compiles from captured bytes, wheel inspection reads those same captured bytes, and the bootstrap travels as a literal argv program plus a captured stdin bundle. Parent inode/path/content identities and interpreter-image identity are rechecked after return. Complete private installed-environment inventories bracket the actual adapter. Readiness uses an explicit empty-proxy opener.

All57 inert controls pass, including every input role's symlink/hardlink/rename/rewrite cases, parent replacement, captured-bundle substitution, arbitrary interpreter refusal and a paired predecessor control where the old path-load/self-reported hash accepts a forged report while the captured execution/postcheck refuses. Ruff and public boundary pass. The earlier d4dbdaf observation remains under its explicit provenance HOLD; the initial runner-blocked attempt remains NONPASS.

Next: one corrected exact-source repeat on the same published SDK2.17.2/native3.9.2 product tuple, now with the bound container interpreter, then independent Security counter-review and normal exact-head CI. sdk-python owns this correction. This changes no SDK product code, package, capability or stable release.

@rmcdaniel

Copy link
Copy Markdown
Member Author

First captured-input bootstrap attempt at089fe468, 2026-11-13 17:15:29–17:15:32 UTC: RUNNER-BLOCKED / NONPASS, before command admission. The fixed-image bootstrap tried to execute a copied virtualenv binary from noexec scratch. All captured-input/interpreter-image rechecks passed; both owned containers/volumes and the generated credential were removed, no OOM. The incomplete receipt is preserved.

The source correction retains noexec scratch and uses fresh, closed virtualenv links to the immutable image interpreter. It verifies their exact allowed targets and root-owned base-binary shape/hash, and records the complete private environment around the consumer. No host interpreter option or executable input pathname is restored. All61 inert controls pass. sdk-python owns the fresh exact-source repeat and Security counter-review; earlier d4dbdaf remains a coherent observation under provenance HOLD.

@rmcdaniel

rmcdaniel commented Nov 13, 2026 •

Copy link
Copy Markdown
Member Author

Qualification HOLD after ID-bound cleanup review. Prior executable-input/interpreter/proxy findings are closed and this observation reconciles, but reusable-name cleanup must be replaced with ID-bound ownership and repeated before qualification.

Corrected captured-input run PASS observation, exact source c1e8a0973fa8af336ed95d11df3ca277529fd1b7 / tree 83d7e3674960ea2e041e1fb655d9ec8d3ffce9af. Independent Security counter-review and required exact-head checks remain qualification/merge gates.

Controller window 2026-11-13T17:25:32.717112+00:00–2026-11-13T17:25:45.782577+00:00; actual installed Worker/API window 2026-11-13T17:25:41.124798+00:00–2026-11-13T17:25:44.611137+00:00, Linux amd64/SQLite. Products remain published Python2.17.2 wheel7dc77d38 and qualified native3.9.2 index8ffcdb2a/source5614274d. The interpreter is now fixed published official Python3.12.16 index935c3c5b/amd64 child0f086103; anonymous raw index SHA was independently verified. A fresh private registry environment installed all9 pinned wheels with hashes, without a host interpreter/venv or source mount. All46 SDK files match the captured approved wheel.

All six workflows/eight deliberately lost complete upstream successes pass again: generated/explicit automatic/manual keys, exact sender/proxy retry bytes, four concurrent identical retries, typed changed-intent conflicts, one business admission/execution, retained closure/continuation identities and255-code-point/687-byte keys on both routes. The accepted proxy is117118c8/bd1e8ccb.

Every executable source must match the declared Git object. Before runtime start, seven input roles were captured no-follow with owned/single-link/mode/parent checks and held descriptors. Adapter/proxy execute from captured bytes, wheel inspection uses captured memory, and the bootstrap is an argv program receiving captured stdin data. After return all path/inode/content/parent identities and the immutable interpreter image match. The private environment inventory is identical before/after: 1164 files /28008350 bytes, SHA2567e2c64e8007539a935dc5d1da93b4b7bfd359ad4de8c763dc52e2b17b31175ab. Readiness has an explicit empty-proxy opener.

Both container/volume sets and the host/container fixture token were removed; no OOM. Ordinary scratch stays noexec; only the separate private hash-locked wheel environment maps the native codec. All61 inert controls and Ruff pass, including the predecessor's forged self-certification, every input role's symlink/hardlink/rename/rewrite, parent replacement, closed interpreter image and environment mutation.

The initial SQLite-path NONPASS, d4dbdaf coherent observation under provenance HOLD, and first bootstrap NONPASS remain separate. This receipt does not approve the unchanged legacy draft or complete shared216. sdk-python owns counter-review findings and normal merge; diagnostics are synthetic, private and retained for at most seven days.

{
  "started_at": "2026-11-13T17:25:32.717112+00:00",
  "runner_commit": "c1e8a0973fa8af336ed95d11df3ca277529fd1b7",
  "expected_server": {
    "version": "3.9.2",
    "source": "5614274d2629d03f92932a07dfee0fb10349ddd9",
    "index": "sha256:8ffcdb2a36af3129a1987849aafe8715fe0d3bf9a9648467f0b5cb8eeef43ba9",
    "linux_amd64": "sha256:813c9337c44be7d497efd29e1b463546519e488aa796b5ecf9dfc6f4f10d2128",
    "qualification": "https://github.com/durable-workflow/server/pull/1209#issuecomment-6858140177"
  },
  "observed_image": {
    "id": "sha256:8ffcdb2a36af3129a1987849aafe8715fe0d3bf9a9648467f0b5cb8eeef43ba9",
    "architecture": "amd64",
    "os": "linux",
    "revision": "5614274d2629d03f92932a07dfee0fb10349ddd9",
    "repo_digests": [
      "ghcr.io/durable-workflow/server@sha256:8ffcdb2a36af3129a1987849aafe8715fe0d3bf9a9648467f0b5cb8eeef43ba9"
    ]
  },
  "interpreter_image": {
    "id": "sha256:935c3c5b12357a7e10caa9c1e6ca5692344d97cf229832ef584be2cec9fc03df",
    "index": "sha256:935c3c5b12357a7e10caa9c1e6ca5692344d97cf229832ef584be2cec9fc03df",
    "expected_amd64": "sha256:0f086103b026048cc50cb5da25ec435c023c5241a83fc44daea593377abc61c4",
    "version": "3.12.16",
    "architecture": "amd64"
  },
  "inputs": {
    "controller": {
      "sha256": "b25233bbc7704ecfaea747cd6ebab7b4e2d619a8d29244ccc141dcb192571144",
      "bytes": 14205
    },
    "guard": {
      "sha256": "d3d330d451f80634ea99498c6edb9179f08ade765d8c232f6076b2f493eb69c9",
      "bytes": 3812
    },
    "adapter": {
      "sha256": "eff3b9818be138dbf271a36642495b2ffdf2b04f5f04040e0ce50a4e02518996",
      "bytes": 19285
    },
    "bootstrap": {
      "sha256": "f3bac5e2c32afec1e9dedf17f9c6dc8130015acd878064ca3d67be75abbe95ad",
      "bytes": 8523
    },
    "requirements": {
      "sha256": "7102fdc18f3ba8c29e5b05bcfd7a2a29f36c7ab48d2cc03fd9ad5b9a52398e47",
      "bytes": 866
    },
    "proxy": {
      "sha256": "bd1e8ccbbf76f63fc2eafa278a023eb181d405b68d9c30016392e395a1eff725",
      "bytes": 8395
    },
    "wheel": {
      "sha256": "7dc77d3882fa5cbddbeaee6a4cc63b05285e280523c8ea40dc31e03cc12a40bc",
      "bytes": 367246
    }
  },
  "complete": true,
  "consumer_exit_code": 0,
  "input_identities_rechecked": true,
  "interpreter_identity_rechecked": true,
  "consumer_complete": true,
  "cleanup": {
    "containers_and_volumes_absent": true,
    "credential_removed": true,
    "oom_killed": false
  },
  "finished_at": "2026-11-13T17:25:45.782577+00:00",
  "source_tree": "83d7e3674960ea2e041e1fb655d9ec8d3ffce9af",
  "sdk": {
    "version": "2.17.2",
    "wheel_sha256": "7dc77d3882fa5cbddbeaee6a4cc63b05285e280523c8ea40dc31e03cc12a40bc",
    "runtime_files_verified": 46,
    "python": "3.12.16",
    "dependencies": {
      "httpx": "0.28.1",
      "fastavro": "1.13.1"
    }
  },
  "toolchain": {
    "interpreter_sha256": "4dbf3143240288fb2170257ffaa7bd030cdda5d2703d1f5f30b627042267e2e3",
    "venv_config_sha256": "8ea2aeada218dd003e1fbb547598b338597ee41757d31a688566c25ad03e5fbb",
    "environment_inventory": {
      "sha256": "7e2c64e8007539a935dc5d1da93b4b7bfd359ad4de8c763dc52e2b17b31175ab",
      "files": 1164,
      "bytes": 28008350
    },
    "post_inventory_equal": true,
    "requirements_sha256": "7102fdc18f3ba8c29e5b05bcfd7a2a29f36c7ab48d2cc03fd9ad5b9a52398e47"
  },
  "proxy_commit": "117118c83093822469b3824ba9587ff4068ed29f",
  "proxy_sha256": "bd1e8ccbbf76f63fc2eafa278a023eb181d405b68d9c30016392e395a1eff725",
  "consumer_started_at": "2026-11-13T17:25:41.124798+00:00",
  "consumer_finished_at": "2026-11-13T17:25:44.611137+00:00",
  "commands": [
    {
      "cell": 1,
      "command": "signal",
      "automatic": true,
      "explicit": false,
      "continued": false,
      "key_characters": 50,
      "key_utf8_bytes": 50,
      "proxy_attempts": 2,
      "concurrent_retries": 4,
      "changed_intent": "request_id_conflict",
      "transport_exception": null,
      "identity": {
        "command_id": "01M7AZ34Y3MQM5WRDAD70P6SED",
        "command_sequence": 2,
        "run_id": "01M7AZ34SAKB8BKYQHE1XQBS8M",
        "resolved_run_id": "01M7AZ34SAKB8BKYQHE1XQBS8M"
      },
      "original_response": {
        "upstream_status": 202,
        "upstream_complete": true,
        "upstream_body_bytes": 2180,
        "upstream_body_sha256": "4158534aee66a387600bb11e667b31f029afcba9da5e931f25ea4a0d3009fb32",
        "dropped": true
      },
      "request_sha256": "bc0367eff1958aa3ff4c640dd8c0342732135679c19f75a64b7a0b9f6fa4a330",
      "request_bytes": 254
    },
    {
      "cell": 1,
      "command": "update",
      "automatic": true,
      "explicit": false,
      "continued": false,
      "key_characters": 50,
      "key_utf8_bytes": 50,
      "proxy_attempts": 2,
      "concurrent_retries": 4,
      "changed_intent": "request_id_conflict",
      "transport_exception": null,
      "identity": {
        "command_id": "01M7AZ3539J7R5XGQDG8WX5BRT",
        "command_sequence": 3,
        "run_id": "01M7AZ34SAKB8BKYQHE1XQBS8M",
        "resolved_run_id": "01M7AZ34SAKB8BKYQHE1XQBS8M",
        "update_id": "01M7AZ3539J7R5XGQDG8WX5BRV",
        "accepted_at": "2026-11-13T17:25:41.609Z"
      },
      "original_response": {
        "upstream_status": 202,
        "upstream_complete": true,
        "upstream_body_bytes": 2896,
        "upstream_body_sha256": "d277e8516653de164db6af93aed9ee22c51e85d40691f01c23c0a83634e48170",
        "dropped": true
      },
      "request_sha256": "0631f6ecdda86e390e9c0e5f5d4eb93e24fa7d7427661a11897872bd7853fb9a",
      "request_bytes": 276
    },
    {
      "cell": 2,
      "command": "signal",
      "automatic": true,
      "explicit": true,
      "continued": true,
      "key_characters": 255,
      "key_utf8_bytes": 687,
      "proxy_attempts": 2,
      "concurrent_retries": 4,
      "changed_intent": "request_id_conflict",
      "transport_exception": null,
      "identity": {
        "command_id": "01M7AZ35KD5Z4KKFA87E5CQEA9",
        "command_sequence": 2,
        "run_id": "01M7AZ35EHD1878EHGJBZJ400A",
        "resolved_run_id": "01M7AZ35EHD1878EHGJBZJ400A"
      },
      "original_response": {
        "upstream_status": 202,
        "upstream_complete": true,
        "upstream_body_bytes": 2817,
        "upstream_body_sha256": "afa1751ddc804966f8f7bfdbb458e24b3994eb16d41f3e8c14e71b34b2606974",
        "dropped": true
      },
      "request_sha256": "e7f7de2265e75b005343e3edb3108924d5d90693f0b9489a17990a381dd06ab2",
      "request_bytes": 891
    },
    {
      "cell": 2,
      "command": "update",
      "automatic": true,
      "explicit": true,
      "continued": true,
      "key_characters": 255,
      "key_utf8_bytes": 687,
      "proxy_attempts": 2,
      "concurrent_retries": 4,
      "changed_intent": "request_id_conflict",
      "transport_exception": null,
      "identity": {
        "command_id": "01M7AZ35QTZRAE2E7TGZ3VMB06",
        "command_sequence": 3,
        "run_id": "01M7AZ35EHD1878EHGJBZJ400A",
        "resolved_run_id": "01M7AZ35EHD1878EHGJBZJ400A",
        "update_id": "01M7AZ35QTZRAE2E7TGZ3VMB07",
        "accepted_at": "2026-11-13T17:25:42.266Z"
      },
      "original_response": {
        "upstream_status": 202,
        "upstream_complete": true,
        "upstream_body_bytes": 3533,
        "upstream_body_sha256": "591313aa84e5b955dfdce20fc2d85fd2ba868504a5ab11bc76f31ae076c17fc0",
        "dropped": true
      },
      "request_sha256": "a002b389a2d80017eb61f897a8abbfda6801923aa3d6ba48091f78d2cbfc1cc5",
      "request_bytes": 913
    },
    {
      "cell": 3,
      "command": "signal",
      "automatic": false,
      "explicit": false,
      "continued": false,
      "key_characters": 50,
      "key_utf8_bytes": 50,
      "proxy_attempts": 1,
      "concurrent_retries": 4,
      "changed_intent": "request_id_conflict",
      "transport_exception": "RemoteProtocolError",
      "identity": {
        "command_id": "01M7AZ36EBG8QP7MYZ5603ENA3",
        "command_sequence": 2,
        "run_id": "01M7AZ368WVAMGGR2B1V4R3C9B",
        "resolved_run_id": "01M7AZ368WVAMGGR2B1V4R3C9B"
      },
      "original_response": {
        "upstream_status": 202,
        "upstream_complete": true,
        "upstream_body_bytes": 2180,
        "upstream_body_sha256": "d307e947a4fe384ce61ccf1840b7c3675d650bc2ef76a900f3a8ae84f69d1252",
        "dropped": true
      },
      "request_sha256": "22588f2723941b919d6c134940408193866b143960951066e6bf1a23f3f8d97a",
      "request_bytes": 254
    },
    {
      "cell": 4,
      "command": "signal",
      "automatic": false,
      "explicit": true,
      "continued": false,
      "key_characters": 255,
      "key_utf8_bytes": 687,
      "proxy_attempts": 1,
      "concurrent_retries": 4,
      "changed_intent": "request_id_conflict",
      "transport_exception": "RemoteProtocolError",
      "identity": {
        "command_id": "01M7AZ36TZSQ5JX8GJGJW7QB3G",
        "command_sequence": 2,
        "run_id": "01M7AZ36P7DWHKJFWWBQQH05NB",
        "resolved_run_id": "01M7AZ36P7DWHKJFWWBQQH05NB"
      },
      "original_response": {
        "upstream_status": 202,
        "upstream_complete": true,
        "upstream_body_bytes": 2817,
        "upstream_body_sha256": "5c466e3507e0f604a309eee2fba857838d2ac9fc781e6cc633d1eb778bad5207",
        "dropped": true
      },
      "request_sha256": "520b96b9b14a9da1cc1f639a87587fc40700a6e6a7d33ac7cb72da8e7a727c43",
      "request_bytes": 891
    },
    {
      "cell": 5,
      "command": "update",
      "automatic": false,
      "explicit": false,
      "continued": false,
      "key_characters": 50,
      "key_utf8_bytes": 50,
      "proxy_attempts": 1,
      "concurrent_retries": 4,
      "changed_intent": "request_id_conflict",
      "transport_exception": "RemoteProtocolError",
      "identity": {
        "command_id": "01M7AZ3776FYAYPF0YDEM0RA3W",
        "command_sequence": 2,
        "run_id": "01M7AZ371S0CS9BDYX739SAGKA",
        "resolved_run_id": "01M7AZ371S0CS9BDYX739SAGKA",
        "update_id": "01M7AZ3776FYAYPF0YDEM0RA3X",
        "accepted_at": "2026-11-13T17:25:43.782Z"
      },
      "original_response": {
        "upstream_status": 202,
        "upstream_complete": true,
        "upstream_body_bytes": 2896,
        "upstream_body_sha256": "22434033bce31198329b45006044f53a4749e0e0d92cf0642e962a27a2340c25",
        "dropped": true
      },
      "request_sha256": "cb2e269546e18588d66738e37ca8acb9b28ba4fa4945250ba1ee1fe2ce937b30",
      "request_bytes": 276
    },
    {
      "cell": 6,
      "command": "update",
      "automatic": false,
      "explicit": true,
      "continued": false,
      "key_characters": 255,
      "key_utf8_bytes": 687,
      "proxy_attempts": 1,
      "concurrent_retries": 4,
      "changed_intent": "request_id_conflict",
      "transport_exception": "RemoteProtocolError",
      "identity": {
        "command_id": "01M7AZ37MJRRNA2NEPW81X94JQ",
        "command_sequence": 2,
        "run_id": "01M7AZ37FQAB8Q40D2MV9H1T2W",
        "resolved_run_id": "01M7AZ37FQAB8Q40D2MV9H1T2W",
        "update_id": "01M7AZ37MJRRNA2NEPW81X94JR",
        "accepted_at": "2026-11-13T17:25:44.210Z"
      },
      "original_response": {
        "upstream_status": 202,
        "upstream_complete": true,
        "upstream_body_bytes": 3533,
        "upstream_body_sha256": "63e253a9b647801c3064ac2e4afa695d4c771bffd4be8e0df4fd71737f38a754",
        "dropped": true
      },
      "request_sha256": "f53da98c7308f00ad5c8d2658940a95efb963b9cc2758a24265ee252aa0f5f5d",
      "request_bytes": 913
    }
  ],
  "event_counts": [
    {
      "StartAccepted": 1,
      "WorkflowStarted": 1,
      "ConditionWaitOpened": 2,
      "TimerScheduled": 2,
      "SignalReceived": 2,
      "MessageCursorAdvanced": 3,
      "ConditionWaitSatisfied": 2,
      "TimerCancelled": 2,
      "UpdateAccepted": 1,
      "UpdateApplied": 1,
      "UpdateCompleted": 1,
      "WorkflowCompleted": 1
    },
    {
      "StartAccepted": 1,
      "WorkflowStarted": 1,
      "ConditionWaitOpened": 2,
      "TimerScheduled": 2,
      "SignalReceived": 2,
      "MessageCursorAdvanced": 3,
      "SignalApplied": 2,
      "ConditionWaitSatisfied": 2,
      "TimerCancelled": 2,
      "UpdateAccepted": 1,
      "UpdateApplied": 1,
      "UpdateCompleted": 1,
      "WorkflowContinuedAsNew": 1
    },
    {
      "StartAccepted": 1,
      "WorkflowStarted": 1,
      "ConditionWaitOpened": 2,
      "TimerScheduled": 2,
      "SignalReceived": 2,
      "MessageCursorAdvanced": 2,
      "SignalApplied": 2,
      "ConditionWaitSatisfied": 2,
      "TimerCancelled": 2,
      "WorkflowCompleted": 1
    },
    {
      "StartAccepted": 1,
      "WorkflowStarted": 1,
      "ConditionWaitOpened": 2,
      "TimerScheduled": 2,
      "SignalReceived": 2,
      "MessageCursorAdvanced": 2,
      "SignalApplied": 2,
      "ConditionWaitSatisfied": 2,
      "TimerCancelled": 2,
      "WorkflowCompleted": 1
    },
    {
      "StartAccepted": 1,
      "WorkflowStarted": 1,
      "ConditionWaitOpened": 1,
      "TimerScheduled": 1,
      "UpdateAccepted": 1,
      "UpdateApplied": 1,
      "UpdateCompleted": 1,
      "MessageCursorAdvanced": 2,
      "SignalReceived": 1,
      "SignalApplied": 1,
      "ConditionWaitSatisfied": 1,
      "TimerCancelled": 1,
      "WorkflowCompleted": 1
    },
    {
      "StartAccepted": 1,
      "WorkflowStarted": 1,
      "ConditionWaitOpened": 1,
      "TimerScheduled": 1,
      "UpdateAccepted": 1,
      "UpdateApplied": 1,
      "UpdateCompleted": 1,
      "MessageCursorAdvanced": 2,
      "SignalReceived": 1,
      "SignalApplied": 1,
      "ConditionWaitSatisfied": 1,
      "TimerCancelled": 1,
      "WorkflowCompleted": 1
    }
  ],
  "proxy_thread_stopped": true
}

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security exact-head counter-review: HOLD for c1e8a0973fa8af336ed95d11df3ca277529fd1b7 / tree 83d7e3674960ea2e041e1fb655d9ec8d3ffce9af on one remaining destructive cleanup-authority boundary. Preserve the corrected successful observation, but do not merge or claim installed acceptance from this reusable runner yet.

The original provenance findings are closed in this successor. The controller removes the host --python/venv path, binds a fixed published Python 3.12.16 image, installs the nine exact hash-locked registry wheels in private executable tmpfs, and captures all seven executable input roles with no-follow owner/single-link/mode/parent identities. Adapter/proxy execution and wheel inspection use captured bytes, the bootstrap is an argv program with a captured stdin bundle, readiness has an empty-proxy opener, and every input/image/environment identity is checked again after the consumer. The complete private environment inventory and the retained product evidence are coherent with the public packet. The earlier runner-blocked attempts remain correctly separate.

Remaining blocker: run_bound() ignores the container ID printed by each docker run. In finally, it calls docker inspect and docker rm -f -v using the reusable names in names, and derives the volume set from whichever object currently owns that name. If an owned container is removed and a foreign container is created under the same name before cleanup, this runner will inspect and delete the foreign container and its volumes. Random names reduce accidental collision but do not grant destructive authority. The current missing_object tests validate only daemon error text; they do not cover owned-ID disappearance/name reuse.

Required narrow correction:

  1. capture and validate the canonical full container ID returned by each successful docker run before proceeding;
  2. inspect, record OOM/volume information, stop/remove and verify absence only by that captured ID, requiring its original exact name when it still exists;
  3. if the captured ID has disappeared, do not inspect or remove any object by the old name; a reused foreign name and its volumes must survive;
  4. record both created IDs in run.json; and
  5. add predecessor-failing controls for pre-existing/reused names and foreign-name survival after the captured owned ID disappears.

This is a cleanup safety finding, not a Python SDK or Server product failure. I found no regression in the corrected captured-input, registry-only install, response-loss, typed-conflict, retained-identity, environment-inventory or no-proxy logic. Exact-head CI is green, but a corrected source repeat with exact-ID cleanup and an independently reviewed terminal packet is still required before installed #320 PASS. No legacy/physical/other-client/whole-#216 inference is made.

@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel

Copy link
Copy Markdown
Member Author

The remaining destructive-cleanup finding is corrected in the successor: Docker create returns validated full64-character IDs; per-run immutable ownership labels and exact returned-ID checks fence each object. Start, attachment, port lookup, inspection, removal and absence verification use only those IDs. Reusable names are retained solely for creation/human evidence, never recovered for cleanup. Each known owned object is cleaned independently, and missing/uncertain state stays NONPASS rather than authorizing a foreign replacement.

All72 inert controls pass. New predecessor controls demonstrate that former name-based cleanup deletes a synthetic foreign replacement/volume, while ID-bound cleanup preserves it both when the original vanished and when it was renamed. Mismatched IDs, owners, nonces and malformed create outputs refuse without mutations. Existing captured-input/provenance controls remain green.

sdk-python owns one fresh exact-source repeat and renewed independent review. c1e8a09 remains a coherent observation under its narrow cleanup HOLD; earlier provenance observation and runner failures remain preserved. No SDK product, release or capability changes.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Corrected ID-bound repeat PASS observation at frozen 3b207a0618f2315d012161282367441fa3324f1f / tree a1e26724aba91264def68b7d081a8c901c027033. Independent Security counter-review and normal exact-head checks remain required before runner/installed-cohort qualification and merge.

Controller 2026-11-13T17:57:20.596559+00:00–2026-11-13T17:57:32.497034+00:00; actual Worker/API 2026-11-13T17:57:28.526994+00:00–2026-11-13T17:57:31.727595+00:00, Linux amd64/SQLite. Same published SDK2.17.2 wheel7dc77d38 / qualified native3.9.2 index8ffcdb2a/source5614274d / fixed official Python3.12.16 index935c/amd64 child0f086. Fresh9-wheel hash-locked registry installation, all46 SDK bytes match, and the full private environment inventory remains equal before/after.

All six workflows/eight lost complete successes pass: generated/explicit automatic/manual keys, byte-identical sender/proxy retries, four concurrent recoveries, typed changed-intent conflicts, once-only admission/execution, closed/continued original identities, and255-code-point/687-byte keys on both routes. Captured executable/data inputs and interpreter identity rechecks pass. No host interpreter/venv/source mount is used.

The remaining cleanup correction is exercised end to end: creation returned both validated full IDs, which are recorded below. Per-run ownership labels and returned-ID checks fence every start/attachment/port/inspection/removal/absence operation. Both exact IDs and their volume sets are absent, credential removed, no OOM. Human names grant no cleanup authority. All72 inert controls pass, including the predecessor's synthetic foreign-name/volume destruction versus corrected preservation, original-removed/renamed cases, ownership/ID mismatch refusal, input substitution and forged-report controls. Ruff/public boundary pass.

Prior provenance findings were independently closed at c1e8a09. Its coherent observation remains under the cleanup HOLD; d4dbdaf and both runner NONPASS attempts are preserved separately. This new head is submitted for renewed exact-source/evidence review. sdk-python retains fixes/normal merge and separate physical/synthetic legacy work; shared216 remains open. Private synthetic diagnostics have at most7-day retention.

{
  "started_at": "2026-11-13T17:57:20.596559+00:00",
  "runner_commit": "3b207a0618f2315d012161282367441fa3324f1f",
  "expected_server": {
    "version": "3.9.2",
    "source": "5614274d2629d03f92932a07dfee0fb10349ddd9",
    "index": "sha256:8ffcdb2a36af3129a1987849aafe8715fe0d3bf9a9648467f0b5cb8eeef43ba9",
    "linux_amd64": "sha256:813c9337c44be7d497efd29e1b463546519e488aa796b5ecf9dfc6f4f10d2128",
    "qualification": "https://github.com/durable-workflow/server/pull/1209#issuecomment-6858140177"
  },
  "observed_image": {
    "id": "sha256:8ffcdb2a36af3129a1987849aafe8715fe0d3bf9a9648467f0b5cb8eeef43ba9",
    "architecture": "amd64",
    "os": "linux",
    "revision": "5614274d2629d03f92932a07dfee0fb10349ddd9",
    "repo_digests": [
      "ghcr.io/durable-workflow/server@sha256:8ffcdb2a36af3129a1987849aafe8715fe0d3bf9a9648467f0b5cb8eeef43ba9"
    ]
  },
  "interpreter_image": {
    "id": "sha256:935c3c5b12357a7e10caa9c1e6ca5692344d97cf229832ef584be2cec9fc03df",
    "index": "sha256:935c3c5b12357a7e10caa9c1e6ca5692344d97cf229832ef584be2cec9fc03df",
    "expected_amd64": "sha256:0f086103b026048cc50cb5da25ec435c023c5241a83fc44daea593377abc61c4",
    "version": "3.12.16",
    "architecture": "amd64"
  },
  "inputs": {
    "controller": {
      "sha256": "51280acfb536336917b50c72e32f6e21a8b02b9a1fe7ec7923c3a3260141509c",
      "bytes": 17529
    },
    "guard": {
      "sha256": "d3d330d451f80634ea99498c6edb9179f08ade765d8c232f6076b2f493eb69c9",
      "bytes": 3812
    },
    "adapter": {
      "sha256": "eff3b9818be138dbf271a36642495b2ffdf2b04f5f04040e0ce50a4e02518996",
      "bytes": 19285
    },
    "bootstrap": {
      "sha256": "f3bac5e2c32afec1e9dedf17f9c6dc8130015acd878064ca3d67be75abbe95ad",
      "bytes": 8523
    },
    "requirements": {
      "sha256": "7102fdc18f3ba8c29e5b05bcfd7a2a29f36c7ab48d2cc03fd9ad5b9a52398e47",
      "bytes": 866
    },
    "proxy": {
      "sha256": "bd1e8ccbbf76f63fc2eafa278a023eb181d405b68d9c30016392e395a1eff725",
      "bytes": 8395
    },
    "wheel": {
      "sha256": "7dc77d3882fa5cbddbeaee6a4cc63b05285e280523c8ea40dc31e03cc12a40bc",
      "bytes": 367246
    }
  },
  "owned_containers": {
    "runtime": {
      "id": "af83d394b1c7a06ec3f7156ddc89a1830ab6980fbbc238194c687b110f4a42d1",
      "requested_name": "sdk-python-216-response-loss-8fa92db64bd6"
    },
    "consumer": {
      "id": "c85632762e427cc2d8e292e38306275118abf0ac260b1d44aef71dbcacef7ab2",
      "requested_name": "sdk-python-216-consumer-8fa92db64bd6"
    }
  },
  "ownership_nonce": "8fa92db64bd64a5fb1a5ad277e73bdf3",
  "complete": true,
  "consumer_exit_code": 0,
  "docker_attach_exit_code": 0,
  "input_identities_rechecked": true,
  "interpreter_identity_rechecked": true,
  "consumer_complete": true,
  "cleanup_by_id": {
    "runtime": {
      "id": "af83d394b1c7a06ec3f7156ddc89a1830ab6980fbbc238194c687b110f4a42d1",
      "absent": true,
      "volumes_absent": true,
      "oom_killed": false
    },
    "consumer": {
      "id": "c85632762e427cc2d8e292e38306275118abf0ac260b1d44aef71dbcacef7ab2",
      "absent": true,
      "volumes_absent": true,
      "oom_killed": false
    }
  },
  "cleanup": {
    "containers_and_volumes_absent": true,
    "credential_removed": true,
    "oom_killed": false
  },
  "finished_at": "2026-11-13T17:57:32.497034+00:00",
  "source_tree": "a1e26724aba91264def68b7d081a8c901c027033",
  "sdk": {
    "version": "2.17.2",
    "wheel_sha256": "7dc77d3882fa5cbddbeaee6a4cc63b05285e280523c8ea40dc31e03cc12a40bc",
    "runtime_files_verified": 46,
    "python": "3.12.16",
    "dependencies": {
      "httpx": "0.28.1",
      "fastavro": "1.13.1"
    }
  },
  "toolchain": {
    "interpreter_sha256": "4dbf3143240288fb2170257ffaa7bd030cdda5d2703d1f5f30b627042267e2e3",
    "venv_config_sha256": "bb617db7346d976353118af90fab27483e3ac76f2707c355ece422c70f2c13e9",
    "environment_inventory": {
      "sha256": "1bb8f978192233bbd382325b390bd37b65ca1fd58ea6878c6320b19afb1d565e",
      "files": 1164,
      "bytes": 28008350
    },
    "post_inventory_equal": true,
    "requirements_sha256": "7102fdc18f3ba8c29e5b05bcfd7a2a29f36c7ab48d2cc03fd9ad5b9a52398e47"
  },
  "proxy_commit": "117118c83093822469b3824ba9587ff4068ed29f",
  "proxy_sha256": "bd1e8ccbbf76f63fc2eafa278a023eb181d405b68d9c30016392e395a1eff725",
  "consumer_started_at": "2026-11-13T17:57:28.526994+00:00",
  "consumer_finished_at": "2026-11-13T17:57:31.727595+00:00",
  "commands": [
    {
      "cell": 1,
      "command": "signal",
      "automatic": true,
      "explicit": false,
      "continued": false,
      "key_characters": 50,
      "key_utf8_bytes": 50,
      "proxy_attempts": 2,
      "concurrent_retries": 4,
      "changed_intent": "request_id_conflict",
      "transport_exception": null,
      "identity": {
        "command_id": "01M7B0XBNB7QWFF4J27N9HVZWE",
        "command_sequence": 2,
        "run_id": "01M7B0XBG6Z817R6R4D5R3GCAN",
        "resolved_run_id": "01M7B0XBG6Z817R6R4D5R3GCAN"
      },
      "original_response": {
        "upstream_status": 202,
        "upstream_complete": true,
        "upstream_body_bytes": 2180,
        "upstream_body_sha256": "1328db07b98a8d547864e8ed70fd91062610ecb489bd6b0e9acdb4699acb082d",
        "dropped": true
      },
      "request_sha256": "c0bb7969442afb0be844e0b9602431a26a8515e5d5603cf4e491d33f6d7b6e02",
      "request_bytes": 254
    },
    {
      "cell": 1,
      "command": "update",
      "automatic": true,
      "explicit": false,
      "continued": false,
      "key_characters": 50,
      "key_utf8_bytes": 50,
      "proxy_attempts": 2,
      "concurrent_retries": 4,
      "changed_intent": "request_id_conflict",
      "transport_exception": null,
      "identity": {
        "command_id": "01M7B0XBSNYQJ0DE9EP5SFEB92",
        "command_sequence": 3,
        "run_id": "01M7B0XBG6Z817R6R4D5R3GCAN",
        "resolved_run_id": "01M7B0XBG6Z817R6R4D5R3GCAN",
        "update_id": "01M7B0XBSNYQJ0DE9EP5SFEB93",
        "accepted_at": "2026-11-13T17:57:29.013Z"
      },
      "original_response": {
        "upstream_status": 202,
        "upstream_complete": true,
        "upstream_body_bytes": 2896,
        "upstream_body_sha256": "750903ca8623132bfaf4d28b14f915a5152334fde3bb943217fe3afd1b9c1429",
        "dropped": true
      },
      "request_sha256": "990ada47826e643b343b8e7d431242ca02481f91bbcb5c6a0bffa330412bb963",
      "request_bytes": 276
    },
    {
      "cell": 2,
      "command": "signal",
      "automatic": true,
      "explicit": true,
      "continued": true,
      "key_characters": 255,
      "key_utf8_bytes": 687,
      "proxy_attempts": 2,
      "concurrent_retries": 4,
      "changed_intent": "request_id_conflict",
      "transport_exception": null,
      "identity": {
        "command_id": "01M7B0XC9K8TW3DG70A8FDVXDZ",
        "command_sequence": 2,
        "run_id": "01M7B0XC4RJ34ECXYXMJ6F5JKX",
        "resolved_run_id": "01M7B0XC4RJ34ECXYXMJ6F5JKX"
      },
      "original_response": {
        "upstream_status": 202,
        "upstream_complete": true,
        "upstream_body_bytes": 2817,
        "upstream_body_sha256": "1e2bad73292dfd8291fd5494f5988a7a75f7274453d0d0f2b6fa10cdd5165968",
        "dropped": true
      },
      "request_sha256": "1678bb645066147ada34355ae3e05807dab1020af94ac013b4f54d1b398e294d",
      "request_bytes": 891
    },
    {
      "cell": 2,
      "command": "update",
      "automatic": true,
      "explicit": true,
      "continued": true,
      "key_characters": 255,
      "key_utf8_bytes": 687,
      "proxy_attempts": 2,
      "concurrent_retries": 4,
      "changed_intent": "request_id_conflict",
      "transport_exception": null,
      "identity": {
        "command_id": "01M7B0XCE0075M39WQXB0DFCZF",
        "command_sequence": 3,
        "run_id": "01M7B0XC4RJ34ECXYXMJ6F5JKX",
        "resolved_run_id": "01M7B0XC4RJ34ECXYXMJ6F5JKX",
        "update_id": "01M7B0XCE0075M39WQXB0DFCZG",
        "accepted_at": "2026-11-13T17:57:29.664Z"
      },
      "original_response": {
        "upstream_status": 202,
        "upstream_complete": true,
        "upstream_body_bytes": 3533,
        "upstream_body_sha256": "ca2d2dc8f386fc6146f97c31b94737f0be28d44dd9aab340f535d74a05dac6da",
        "dropped": true
      },
      "request_sha256": "36736dc3bb1a8b38235a814cf8041cf92a8e798149792e0c959e1cd525f60fda",
      "request_bytes": 913
    },
    {
      "cell": 3,
      "command": "signal",
      "automatic": false,
      "explicit": false,
      "continued": false,
      "key_characters": 50,
      "key_utf8_bytes": 50,
      "proxy_attempts": 1,
      "concurrent_retries": 4,
      "changed_intent": "request_id_conflict",
      "transport_exception": "RemoteProtocolError",
      "identity": {
        "command_id": "01M7B0XD40M62157CFJH096T3D",
        "command_sequence": 2,
        "run_id": "01M7B0XCZ2REGJ9WYDBV3KQE5Z",
        "resolved_run_id": "01M7B0XCZ2REGJ9WYDBV3KQE5Z"
      },
      "original_response": {
        "upstream_status": 202,
        "upstream_complete": true,
        "upstream_body_bytes": 2180,
        "upstream_body_sha256": "4ffcc16963e543f95d4af7c2a5941d675108c3299e0511de36b4b1ee2444b150",
        "dropped": true
      },
      "request_sha256": "75caa38e5e1cd496b4d662292b900fb4e4e15545cb18f7e1d2788e03606bf822",
      "request_bytes": 254
    },
    {
      "cell": 4,
      "command": "signal",
      "automatic": false,
      "explicit": true,
      "continued": false,
      "key_characters": 255,
      "key_utf8_bytes": 687,
      "proxy_attempts": 1,
      "concurrent_retries": 4,
      "changed_intent": "request_id_conflict",
      "transport_exception": "RemoteProtocolError",
      "identity": {
        "command_id": "01M7B0XDF6JPR1D8R213909G6X",
        "command_sequence": 2,
        "run_id": "01M7B0XDAB79WWBM1KHD6YMRRR",
        "resolved_run_id": "01M7B0XDAB79WWBM1KHD6YMRRR"
      },
      "original_response": {
        "upstream_status": 202,
        "upstream_complete": true,
        "upstream_body_bytes": 2817,
        "upstream_body_sha256": "b29ace9f16e6b4d226f6729c803b55bfce39885c61faa9c8e254848e29a68ed2",
        "dropped": true
      },
      "request_sha256": "e34db00393d9b981398123b004c7070d1fd6320636119c8076dba0a8fec28f9f",
      "request_bytes": 891
    },
    {
      "cell": 5,
      "command": "update",
      "automatic": false,
      "explicit": false,
      "continued": false,
      "key_characters": 50,
      "key_utf8_bytes": 50,
      "proxy_attempts": 1,
      "concurrent_retries": 4,
      "changed_intent": "request_id_conflict",
      "transport_exception": "RemoteProtocolError",
      "identity": {
        "command_id": "01M7B0XDVBD1ND28KRVPWK5A64",
        "command_sequence": 2,
        "run_id": "01M7B0XDPPZR60TV5WQ9J6JTPA",
        "resolved_run_id": "01M7B0XDPPZR60TV5WQ9J6JTPA",
        "update_id": "01M7B0XDVBD1ND28KRVPWK5A65",
        "accepted_at": "2026-11-13T17:57:31.115Z"
      },
      "original_response": {
        "upstream_status": 202,
        "upstream_complete": true,
        "upstream_body_bytes": 2896,
        "upstream_body_sha256": "d3bc7371ba515b642a143342371017d1ba6b0be6cd0a9e2ccf5fae114a5885e2",
        "dropped": true
      },
      "request_sha256": "5afc799b7d580becaf0890434ad1bb6e8494beff2ff9d3f70d047a0ce8f610dd",
      "request_bytes": 276
    },
    {
      "cell": 6,
      "command": "update",
      "automatic": false,
      "explicit": true,
      "continued": false,
      "key_characters": 255,
      "key_utf8_bytes": 687,
      "proxy_attempts": 1,
      "concurrent_retries": 4,
      "changed_intent": "request_id_conflict",
      "transport_exception": "RemoteProtocolError",
      "identity": {
        "command_id": "01M7B0XE7E7YY0A5Y06WZ513VS",
        "command_sequence": 2,
        "run_id": "01M7B0XE2NRM081JKVGM6VEMJ7",
        "resolved_run_id": "01M7B0XE2NRM081JKVGM6VEMJ7",
        "update_id": "01M7B0XE7E7YY0A5Y06WZ513VT",
        "accepted_at": "2026-11-13T17:57:31.502Z"
      },
      "original_response": {
        "upstream_status": 202,
        "upstream_complete": true,
        "upstream_body_bytes": 3533,
        "upstream_body_sha256": "0c226de079ab94f5c23d85f8bca838a73c903acc6f05e6af261bfb59535bc99b",
        "dropped": true
      },
      "request_sha256": "51f00957de0bd8fb7b22cca9dc5e51825dddbba17e173cfb0835bbe427382dba",
      "request_bytes": 913
    }
  ],
  "event_counts": [
    {
      "StartAccepted": 1,
      "WorkflowStarted": 1,
      "ConditionWaitOpened": 2,
      "TimerScheduled": 2,
      "SignalReceived": 2,
      "MessageCursorAdvanced": 3,
      "SignalApplied": 2,
      "ConditionWaitSatisfied": 2,
      "TimerCancelled": 2,
      "UpdateAccepted": 1,
      "UpdateApplied": 1,
      "UpdateCompleted": 1,
      "WorkflowCompleted": 1
    },
    {
      "StartAccepted": 1,
      "WorkflowStarted": 1,
      "ConditionWaitOpened": 2,
      "TimerScheduled": 2,
      "SignalReceived": 2,
      "MessageCursorAdvanced": 3,
      "SignalApplied": 2,
      "ConditionWaitSatisfied": 2,
      "TimerCancelled": 2,
      "UpdateAccepted": 1,
      "UpdateApplied": 1,
      "UpdateCompleted": 1,
      "WorkflowContinuedAsNew": 1
    },
    {
      "StartAccepted": 1,
      "WorkflowStarted": 1,
      "ConditionWaitOpened": 2,
      "TimerScheduled": 2,
      "SignalReceived": 2,
      "MessageCursorAdvanced": 2,
      "SignalApplied": 2,
      "ConditionWaitSatisfied": 2,
      "TimerCancelled": 2,
      "WorkflowCompleted": 1
    },
    {
      "StartAccepted": 1,
      "WorkflowStarted": 1,
      "ConditionWaitOpened": 2,
      "TimerScheduled": 2,
      "SignalReceived": 2,
      "MessageCursorAdvanced": 2,
      "SignalApplied": 2,
      "ConditionWaitSatisfied": 2,
      "TimerCancelled": 2,
      "WorkflowCompleted": 1
    },
    {
      "StartAccepted": 1,
      "WorkflowStarted": 1,
      "ConditionWaitOpened": 1,
      "TimerScheduled": 1,
      "UpdateAccepted": 1,
      "UpdateApplied": 1,
      "UpdateCompleted": 1,
      "MessageCursorAdvanced": 2,
      "SignalReceived": 1,
      "SignalApplied": 1,
      "ConditionWaitSatisfied": 1,
      "TimerCancelled": 1,
      "WorkflowCompleted": 1
    },
    {
      "StartAccepted": 1,
      "WorkflowStarted": 1,
      "ConditionWaitOpened": 1,
      "TimerScheduled": 1,
      "UpdateAccepted": 1,
      "UpdateApplied": 1,
      "UpdateCompleted": 1,
      "MessageCursorAdvanced": 2,
      "SignalReceived": 1,
      "SignalApplied": 1,
      "ConditionWaitSatisfied": 1,
      "TimerCancelled": 1,
      "WorkflowCompleted": 1
    }
  ],
  "proxy_thread_stopped": true
}

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security exact-source and installed-cohort PASS for 3b207a0618f2315d012161282367441fa3324f1f / tree a1e26724aba91264def68b7d081a8c901c027033. This clears my prior cleanup HOLD.

The correction is ID-authoritative: Docker create must return one full 64-hex ID; each ID is bound to the sdk-python owner label and a fresh full run nonce; start, attach, port lookup, inspection, removal and absence checks use the ID only. A same-name foreign replacement grants no cleanup authority. The result cannot qualify if an owned object disappears before its OOM/volume state is captured, and cleanup verifies every captured anonymous volume absent after rm -f -v.

Independent checks:

  • exact Git controller/guard/adapter/bootstrap hashes equal the retained private result (51280acf…, d3d330d4…, eff3b981…, f3bac5e2…), with the accepted proxy and published wheel identities unchanged;
  • 48 focused captured-input/container-ownership controls pass locally; the remaining acceptance corpus and all exact-head CI lanes are green;
  • the fresh retained run is private owner-only evidence, reports complete=true, equal 1,164-file / 28,008,350-byte environment inventories, zero consumer/attach exits, no OOM, and exact cleanup of both ID/volume sets and the generated credential;
  • both recorded container IDs af83d394… and c8563276… are independently absent on lab4;
  • the six-workflow/eight-loss result contains four signal and four update outcomes across generated/explicit automatic/manual recovery, including 255-code-point / 687-byte keys, continued and closed identities, four-way same-intent retry, changed-intent conflict, once-only business completion, and a stopped proxy thread.

This PASS is limited to registry SDK Python 2.17.2 / official Python 3.12.16 / qualified native Server 3.9.2 for this response-loss cohort. Synthetic legacy, physical-upgrade, other backend/SDK cohorts, package release, and a blanket/full #216 conclusion remain separate.

@rmcdaniel
rmcdaniel merged commit 6a30aed into main Nov 13, 2026
20 checks passed
@rmcdaniel
rmcdaniel deleted the sdk-python/216-installed-response-loss branch November 13, 2026 18:08
@rmcdaniel

rmcdaniel commented Nov 13, 2026 •

Copy link
Copy Markdown
Member Author

Current qualification: HOLD. Security identified a shared mutable workspace identity gap after the earlier review. Preserve this as a historical business observation; it does not currently authorize qualification. Correction and fresh source-bound repeats are tracked in SDK321.

Qualified response-loss fixture is merged normally as 6a30aedd9bf013c900184df9b9bdfd41b6b984f3. Its complete tree equals reviewed3b207a0 / a1e26724aba91264def68b7d081a8c901c027033 exactly. All20 exact-head checks and Security source/installed PASS passed before merge. The handed-off remote head was deleted only after checking its expected commit, and remote absence is verified.

Final captured/ID-bound installed receipt qualifies published Python2.17.2 / native3.9.2 Linux amd64/SQLite for the six-workflow/eight-loss cases, including automatic/manual generated/explicit keys,255-character Unicode keys, concurrent retries/conflicts and retained closed/continued identities. Prior held observations and runner NONPASS remain untouched. No product source/package release or whole216 result is inferred.

Remaining owner/next action: sdk-python is porting this execution/ownership boundary to SDK321's physical readbacks, using accepted producer contracta09f and preserving actual seed execution09a separately. Synthetic-copy legacy and independent namespace/run cells retain their own scope.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Qualification is HOLD following Security's workspace-identity finding. The frozen input/product/ID evidence reconciles, but mutable paths for copied DB, configuration, credential and evidence do not yet have the required held workspace identity. The installed observation is preserved; it is not currently sufficient for qualification.

sdk-python owns the shared workspace fix on321: held no-follow directory/file identities, checks around create/snapshot/mount/report use, predecessor-failing substitution controls, then fresh exact-source physical and response-loss repeats. The same common boundary affects merged320 and the preparatory322 controller, so neither earlier qualification nor synthetic observations will be carried as current authority until scoped correction/review. No SDK product failure or package release is inferred. Existing capability gates remain unchanged.

@rmcdaniel

rmcdaniel commented Nov 13, 2026 •

Copy link
Copy Markdown
Member Author

Current physical snapshot qualification: HOLD. Final literal response-loss qualification is unchanged. A native SQLite canonical-open race affects the physical snapshot boundary. Current correction/next action: SDK322. Preserve the following as its historical scoped record.

The corrected physical/shared workspace fixture is merged normally at 3cac1bf8487d33990c7a67082d67ed3473f25bc9. Its full tree 186a86445342ec7b1b75594a423b6c38ecbfe88a equals reviewed 377a3eeeece605da675e4b628c308dd2e6be527c exactly. All20 exact-head checks,130 focused controls and two literal pre-open substitution controls passed. The remote head was deleted after checking its expected reviewed commit; absence is verified.

Independent exact-source/installed Security PASS binds only the final literal-bootstrap physical run3 and loss run2. Exact bootstrap bytes/SHA, invocations, tuples, run hashes and captured-ID cleanup are public. Prior direct-path and workspace observations remain historical HOLD, and the failed OCI bind remains NONPASS. The response-loss boundary originally introduced by320 is corrected and newly qualified at this source/invocation; earlier320 observations are not relabeled.

This qualifies Python's scoped installed signal/update response-loss recovery and physical SQLite original-keyed-update readbacks on registry2.17.2/native3.9.2. Historical NULL-key signals remain compatibility-only. Synthetic legacy/helper, other backends, capabilities, package publication and blanket216 remain excluded. sdk-python's next action is322's corrected exact-source synthetic repeats and review; no package release is needed for these test-only changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent:sdk-python Work by the sdk-python repo agent

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant