Qualify installed command response-loss recovery

scripts/conformance/command_keys/README.md

@@ -0,0 +1,83 @@
+# Installed command-response recovery
+
+This adapter exercises the published Python Client and actual Worker for
+[shared command-key acceptance](https://github.com/durable-workflow/.github/issues/216).
+It creates synthetic workflows and a disposable SQLite runtime. Its transport
+fault comes from the independently reviewed shared loopback fixture.
+
+The controller binds the qualified native image/source/platform from
+[Server's publication receipt](https://github.com/durable-workflow/server/pull/1209#issuecomment-6858140177).
+Its interpreter is the published official Python image index
+`sha256:935c3c5b12357a7e10caa9c1e6ca5692344d97cf229832ef584be2cec9fc03df`,
+amd64 child `sha256:0f086103b026048cc50cb5da25ec435c023c5241a83fc44daea593377abc61c4`.
+It starts a fresh private virtualenv in that image and installs all nine registry
+packages from `consumer-requirements.txt`, with exact wheel-only hashes. The SDK
+wheel is the reviewed published artifact. The interpreter/config and entire
+installed environment are fingerprinted before and after the consumer.
+
+Download the published SDK wheel identified by `WHEEL_SHA256` in the controller.
+Download the shared proxy from
+[`117118c8`](https://github.com/durable-workflow/.github/blob/117118c83093822469b3824ba9587ff4068ed29f/conformance/command-keys/committed_response_loss.py).
+Its SHA-256 is
+`bd1e8ccbbf76f63fc2eafa278a023eb181d405b68d9c30016392e395a1eff725`.
+Store both as regular, single-link, owned files in owner-controlled directories.
+Before the experiment, check host/Docker capacity and pull both exact images
+with an empty Docker config. The controller itself does not pull images.
+
+Run from an exact committed checkout:
+
+```sh
+python3 -I scripts/conformance/command_keys/run_response_loss.py \
+  --runner-commit FULL_REVIEWED_RUNNER_COMMIT \
+  --wheel /path/to/downloaded/durable_workflow-2.17.2-py3-none-any.whl \
+  --proxy /path/to/verified/committed_response_loss.py \
+  --output /path/to/private-disk-scratch/new-run
+```
+
+The controller and executable source inputs must equal that commit's Git blobs.
+It opens inputs without following symlinks, checks owner/mode/single-link/file
+bounds and retains descriptors plus captured bytes. The adapter, bootstrap and
+proxy execute from those bytes. Wheel inspection uses the same captured bytes.
+Pathname, inode, content and parent-directory identities are checked again before
+launch and after return. A substituted program cannot certify its own output by
+reporting the original hash. The host does not supply an interpreter or virtualenv.
+
+The new private output directory must not exist; its parent must already exist.
+One runtime and one consumer are each capped at 1 CPU, 1 GiB and 256 PIDs, with
+read-only roots and `--init`. Bounded tmpfs paths remain within each memory cap.
+The consumer's ordinary scratch is noexec. Its separate private wheel environment
+permits native-extension mapping for the hash-locked codec wheel; it is not a
+host mount and accepts no unverified executable input.
+The runtime publishes one ephemeral loopback port. The consumer uses host
+networking for the numeric loopback-only proxy and has no host mounts. Registry
+installation precedes creation of its synthetic credential file. Its entire
+bootstrap/input bundle travels through stdin; executable files are never reopened
+from host paths. Controller readiness explicitly disables environment proxies.
+The consumer has a ten-minute bound, and cleanup owns both containers, anonymous
+volumes and the generated fixture credential.
+Creation returns full container IDs bound to per-run ownership labels. Start,
+inspection, port lookup, removal and absence verification use those IDs. A reused
+human name grants no cleanup authority over its replacement or volumes.
+
+Six workflows exercise eight intentionally lost complete upstream successes:
+automatic/manual recovery with generated/explicit keys on both routes, including
+continuation. Sender and proxy request bytes agree, four concurrent identical
+retries retain one admission, and changed intent gets a typed conflict. Business
+execution/admission counts and closed/continued original identities are checked.
+Explicit keys contain 255 Unicode code points and exceed 255 UTF-8 bytes. Nested
+typed values include integers, booleans, bytes and Unicode. Query receipts are
+read through their `result` field.
+
+Both `consumer.json` and `run.json` must report complete success, with captured
+input and interpreter rechecks, equal environment inventories, no OOM and exact
+cleanup, before qualification. Record the reviewed runner commit, input hashes,
+tuple, timestamps and sanitized outcomes in GitHub. Keep failed attempts as
+NONPASS and diagnostic captures private and temporary. The earlier host-venv
+observation is retained under its Security provenance HOLD, not reused as the
+corrected runner's qualification.
+
+Synthetic legacy transformations and physical upgrades remain separate cells.
+The accepted common stopped-copy helper is owned by
+[SDK Rust260](https://github.com/durable-workflow/sdk-rust/pull/260); Server1210
+owns the real historical seed. Namespace/run fences remain independently scoped.
+This adapter alone does not complete shared216.

scripts/conformance/command_keys/consumer-requirements.txt

@@ -0,0 +1,9 @@
+anyio==4.16.0 --hash=sha256:c66ae634e0b8228396b801741beb3294326d4b6102f7bba4041c0dfb58c29026
+certifi==2026.10.23 --hash=sha256:e9b54de76e530bb6cf0ccd036787103acb58fd42ba930ea8bf948ddbaa2f61d1
+durable-workflow==2.17.2 --hash=sha256:7dc77d3882fa5cbddbeaee6a4cc63b05285e280523c8ea40dc31e03cc12a40bc
+fastavro==1.13.1 --hash=sha256:2f56a127d71e45083306d2650efff827cad0f4b0744dd42cb69c631d77943b1d
+h11==0.16.0 --hash=sha256:63cf8bbe7522de3bf65932fda1d9c2772064ffb3dae62d55932da54b31cb6c86
+httpcore==1.0.9 --hash=sha256:2d400746a40668fc9dec9810239072b40b4484b640a8c38fd654a024c7a1bf55
+httpx==0.28.1 --hash=sha256:d909fcccc110f8c7faf814ca82a9a4d816bc5a6dbfea25d6591d6985b8ba59ad
+idna==3.20 --hash=sha256:ab7ae7122974553370f0bdb919e1a960b2cd1bc1ef0276416d896db81c14582c
+typing_extensions==4.16.0 --hash=sha256:481caa481374e813c1b176ada14e97f1f67a4539ce9cfeb3f350d78d6370c2e8

scripts/conformance/command_keys/consumer_bootstrap.py

@@ -0,0 +1,219 @@
+"""Trusted argv bootstrap inside the fixed published Python image.
+
+No host interpreter, virtualenv, source directory or credential directory is
+mounted. The controller sends only already captured inputs and synthetic data.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import base64
+import hashlib
+import json
+import linecache
+import os
+import stat
+import subprocess
+import sys
+import tempfile
+import venv
+from pathlib import Path
+from types import ModuleType, SimpleNamespace
+
+RESULT_PREFIX = "SDK_PYTHON_CONSUMER_RESULT="
+ROLES = {"adapter", "bootstrap", "proxy", "wheel", "requirements"}
+MAX_BUNDLE_BYTES = 8 * 1024 * 1024
+IMAGE_INTERPRETER = Path("/usr/local/bin/python3.12")
+
+
+def approved_environment_link(relative, target):
+    return (
+        (relative == "lib64" and target == "lib")
+        or (relative in {"bin/python", "bin/python3"} and target == "python3.12")
+        or (relative == "bin/python3.12" and target == str(IMAGE_INTERPRETER))
+    )
+
+
+def decode_inputs(bundle):
+    if set(bundle["sources"]) != ROLES:
+        raise ValueError("captured source roles differ")
+    result = {}
+    for role, entry in bundle["sources"].items():
+        data = base64.b64decode(entry["base64"], validate=True)
+        if hashlib.sha256(data).hexdigest() != entry["sha256"]:
+            raise ValueError("captured bytes differ: " + role)
+        result[role] = data
+    return result
+
+
+def execute_source(data, name):
+    module = ModuleType(name)
+    module.__file__ = "captured-" + name + ".py"
+    sys.modules[name] = module
+    linecache.cache[module.__file__] = (len(data), None, data.decode().splitlines(keepends=True), module.__file__)
+    # The bytes were captured before runtime start. No executable input path
+    # is reopened or imported through a loader.
+    exec(compile(data, module.__file__, "exec"), module.__dict__)
+    return module
+
+
+def inventory(root):
+    entries = []
+    total = 0
+    for path in sorted(root.rglob("*")):
+        relative = str(path.relative_to(root))
+        info = path.lstat()
+        if stat.S_ISDIR(info.st_mode):
+            continue
+        if path.is_symlink():
+            target = os.readlink(path)
+            if not approved_environment_link(relative, target):
+                raise ValueError("unexpected installed-environment link")
+            entries.append([relative, "symlink", target])
+            continue
+        if not stat.S_ISREG(info.st_mode) or info.st_nlink != 1 or info.st_uid != os.getuid():
+            raise ValueError("unexpected installed-environment file")
+        total += info.st_size
+        if total > 128 * 1024 * 1024 or len(entries) > 10_000:
+            raise ValueError("installed-environment inventory exceeds bound")
+        entries.append([relative, stat.S_IMODE(info.st_mode), hashlib.sha256(path.read_bytes()).hexdigest()])
+    digest = hashlib.sha256(json.dumps(entries, separators=(",", ":")).encode()).hexdigest()
+    return {"sha256": digest, "files": len(entries), "bytes": total}
+
+
+def prepare(bundle, sources):
+    if "prepared" in bundle:
+        raise ValueError("caller may not supply a prepared interpreter")
+    root = Path(tempfile.mkdtemp(prefix="sdk-python-installed-"))
+    environment = Path(tempfile.mkdtemp(prefix="environment-", dir="/opt/sdk-python")) / "venv"
+    # Scratch remains noexec. These fresh internal links may target only the
+    # immutable interpreter in the fixed image, never a host/caller program.
+    venv.EnvBuilder(with_pip=True, symlinks=True).create(environment)
+    lock = root / "requirements.txt"
+    lock.write_bytes(sources["requirements"])
+    lock.chmod(0o400)
+    python = environment / "bin/python"
+    executable = python.lstat()
+    if (
+        not stat.S_ISLNK(executable.st_mode)
+        or executable.st_nlink != 1
+        or executable.st_uid != os.getuid()
+        or python.resolve() != IMAGE_INTERPRETER
+    ):
+        raise ValueError("fresh interpreter identity differs")
+    image_binary = IMAGE_INTERPRETER.lstat()
+    if (
+        not stat.S_ISREG(image_binary.st_mode)
+        or image_binary.st_uid != 0
+        or image_binary.st_nlink != 1
+        or image_binary.st_mode & 0o022
+    ):
+        raise ValueError("immutable image interpreter shape differs")
+    binary_sha256 = hashlib.sha256(python.read_bytes()).hexdigest()
+    if binary_sha256 != hashlib.sha256(IMAGE_INTERPRETER.read_bytes()).hexdigest():
+        raise ValueError("virtualenv interpreter differs from fixed image interpreter")
+    clean_environment = {"PATH": os.defpath, "TMPDIR": str(root), "PIP_CONFIG_FILE": "/dev/null"}
+    subprocess.run(
+        [
+            str(python),
+            "-I",
+            "-B",
+            "-m",
+            "pip",
+            "--isolated",
+            "install",
+            "--no-compile",
+            "--no-cache-dir",
+            "--only-binary=:all:",
+            "--require-hashes",
+            "--disable-pip-version-check",
+            "--progress-bar",
+            "off",
+            "--index-url",
+            "https://pypi.org/simple",
+            "-r",
+            str(lock),
+        ],
+        check=True,
+        env=clean_environment,
+        timeout=180,
+    )
+    before = inventory(environment)
+    bundle["prepared"] = {
+        "root": str(environment),
+        "scratch": str(root),
+        "binary_sha256": binary_sha256,
+        "config_sha256": hashlib.sha256((environment / "pyvenv.cfg").read_bytes()).hexdigest(),
+        "inventory": before,
+    }
+    # Both the child program and its inputs come from the same captured bytes.
+    # The interpreter links here to the immutable image, never supplied
+    # by a host pathname. Its complete environment is checked around the call.
+    result = subprocess.run(
+        [str(python), "-I", "-B", "-c", sources["bootstrap"].decode(), "--prepared"],
+        input=json.dumps(bundle).encode(),
+        env=clean_environment,
+        timeout=300,
+    )
+    if inventory(environment) != before:
+        raise ValueError("installed environment changed")
+    raise SystemExit(result.returncode)
+
+
+def main():
+    raw = sys.stdin.buffer.read(MAX_BUNDLE_BYTES + 1)
+    if len(raw) > MAX_BUNDLE_BYTES:
+        raise ValueError("captured bundle exceeds bound")
+    bundle = json.loads(raw)
+    sources = decode_inputs(bundle)
+    if sys.argv[1:] != ["--prepared"]:
+        prepare(bundle, sources)
+    prepared = bundle["prepared"]
+    environment = Path(prepared["root"])
+    if Path(sys.prefix) != environment or Path(sys.executable).parent != environment / "bin":
+        raise ValueError("consumer did not start in its private registry environment")
+    if (
+        hashlib.sha256(Path(sys.executable).read_bytes()).hexdigest() != prepared["binary_sha256"]
+        or hashlib.sha256((environment / "pyvenv.cfg").read_bytes()).hexdigest() != prepared["config_sha256"]
+    ):
+        raise ValueError("prepared interpreter/config identity changed")
+    before = inventory(environment)
+    if before != prepared["inventory"]:
+        raise ValueError("prepared environment changed before consumer execution")
+    root = Path(prepared["scratch"])
+    credential = root / "fixture.env"
+    descriptor = os.open(credential, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
+    with os.fdopen(descriptor, "w") as stream:
+        stream.write("DW_AUTH_TOKEN=" + bundle["token"] + "\n")
+    adapter = execute_source(sources["adapter"], "installed_command_keys")
+    try:
+        result = asyncio.run(
+            adapter.main(
+                SimpleNamespace(
+                    upstream=bundle["upstream"],
+                    credential_file=credential,
+                    proxy_data=sources["proxy"],
+                    wheel_data=sources["wheel"],
+                    adapter_sha256=bundle["sources"]["adapter"]["sha256"],
+                    server_version=bundle["server_version"],
+                    output=root / "consumer.json",
+                )
+            )
+        )
+    finally:
+        credential.unlink(missing_ok=True)
+    after = inventory(environment)
+    if before != after:
+        raise ValueError("installed environment changed during consumer execution")
+    result["toolchain"] = {
+        "interpreter_sha256": prepared["binary_sha256"],
+        "venv_config_sha256": prepared["config_sha256"],
+        "environment_inventory": before,
+        "post_inventory_equal": True,
+        "requirements_sha256": bundle["sources"]["requirements"]["sha256"],
+    }
+    print(RESULT_PREFIX + json.dumps(result), flush=True)
+
+
+if __name__ == "__main__":
+    main()