diff --git a/conformance/README.md b/conformance/README.md index fc54fc3..cfaa969 100644 --- a/conformance/README.md +++ b/conformance/README.md @@ -14,6 +14,10 @@ action remain visible without creating a second backlog. Run every row against the same exact published Workflow, Waterline, Server, CLI, PHP SDK, Python SDK, and Rust SDK tuple. +The [SDK coverage inventory](sdk-coverage.md) identifies actual runtime +directions, language-specific boundaries and remaining executable gaps for +each public experiment. + | Experiment | Published-artifact runner | | --- | --- | | Activities | `durable-workflow/server`: `scripts/conformance/activities-published-artifacts.sh` | @@ -22,6 +26,7 @@ CLI, PHP SDK, Python SDK, and Rust SDK tuple. | Heartbeats | `durable-workflow/server`: the PHP, Python, and Rust `heartbeats-*-published-artifacts.sh` runners | | Migration | `durable-workflow/server`: `scripts/conformance/migration-published-artifacts.sh` | | Namespaces | `durable-workflow/server`: `scripts/conformance/namespaces-published-artifacts.sh` | +| Nexus | `durable-workflow/server`: `scripts/conformance/nexus-published-artifacts.sh` (supported PHP/Python caller-service directions) | | Polyglot | `durable-workflow/sample-app`: `scripts/polyglot.sh` | | Replay | `durable-workflow/server`: `scripts/conformance/replay-published-artifacts.sh` | | Sagas | `durable-workflow/server`: `scripts/conformance/sagas-published-artifacts.sh` (PHP/Python matrix); `durable-workflow/sample-app`: [`polyglot/sagas/`](https://github.com/durable-workflow/sample-app/blob/main/polyglot/sagas/README.md) (five Rust-involving workflow/compensation directions) | @@ -29,7 +34,7 @@ CLI, PHP SDK, Python SDK, and Rust SDK tuple. | SDK matrix | `durable-workflow/server`: PHP and Python published-artifact runners; `durable-workflow/sample-app`: `scripts/playground rust` | | Search attributes | `durable-workflow/server`: `scripts/conformance/search-attributes-published-artifacts.sh` | | Signals and queries | `durable-workflow/server`: `scripts/conformance/signals-queries-published-artifacts.sh` | -| Timers | `durable-workflow/server`: `scripts/conformance/timers-published-artifacts.sh` (embedded PHP workflow scenarios) | +| Timers | `durable-workflow/server`: `scripts/conformance/timers-published-artifacts.sh` (embedded PHP); `durable-workflow/sample-app`: `scripts/sdk-timers.sh` (PHP/Python/Rust SDK workers) | | Worker versioning | `durable-workflow/server`: `scripts/conformance/worker-versioning-published-artifacts.sh` | | Workflow lifecycle | `durable-workflow/server`: `scripts/conformance/workflow-lifecycle-published-artifacts.sh` | | Workflow updates | `durable-workflow/server`: `scripts/conformance/workflow-updates-published-artifacts.sh` | @@ -99,12 +104,14 @@ The current and published-artifact runner execute timer behavior in the Server image with an embedded PHP workflow. The Python SDK artifact pin does not mean a Python worker ran these scenarios; the runner does not require the PHP or Rust SDK -artifacts. Published PHP, Python, and Rust SDKs expose durable timer APIs, but -their service-mode workflow timer paths are not covered by this runner. Record -those paths as unexecuted until separate published-artifact evidence checks -timer completion, replay/restart, cancellation, and history for each relevant -workflow runtime. A passing embedded timer scenario does not fill those SDK -cells. +artifacts. The separate +[Sample App SDK timer experiment](https://github.com/durable-workflow/sample-app/blob/main/polyglot/timers/README.md) +executes actual PHP, Python and Rust timer workflows. It checks completion, +worker SIGKILL with timer fire during absence and cold replay, Server restart +across the deadline, and cooperative cancellation followed beyond the original +timer due time. Each cell checks persisted history and public status. Report +its twelve scenario outcomes separately. Concurrent timer groups and +timer-bearing application upgrades remain outside that focused experiment. ## Report a run diff --git a/conformance/sdk-coverage.md b/conformance/sdk-coverage.md new file mode 100644 index 0000000..c009dc6 --- /dev/null +++ b/conformance/sdk-coverage.md @@ -0,0 +1,86 @@ +# SDK experiment coverage + +Use this inventory with the [conformance runbook](README.md). It maps existing +public experiments to the language directions they actually execute and the +portable behavior still requiring a runtime check. A row describes executable +scope, not a passing result for an arbitrary release tuple. Results and remaining +work belong to [the experiment audit](https://github.com/durable-workflow/.github/issues/122). + +## Choose the meaningful directions + +- Workflow authoring and durable replay require PHP, Python and Rust workflow + execution. A client starting a workflow does not prove its worker runtime. +- Remote activities and children require all nine supported workflow-to-worker + or parent-to-child directions. Retain the direction on failures and restart + checks as well as the successful result. +- A timer belongs to its workflow runtime. It has no separate language-specific + timer worker, so three workflow languages cover the meaningful directions. +- Schedule creation follows the installed client's API. PHP and Python can + create schedules targeting any of the three workflow languages. The Rust + SDK currently has no schedule creation API. Do not invent a Rust creator. +- Nexus follows advertised support in the exact tuple. The current PHP/Python + caller and service APIs give four meaningful directions. Rust has no Nexus + API. A raw HTTP call from a Rust test process would not prove SDK support. +- Rust supports update handlers and clients but refuses synchronous pre-accept + update validators. Exercise ordinary Rust updates and its explicit capability + refusal. PHP/Python validator execution has its own supported directions. +- Database migration, infrastructure failover and PHP framework integration + each have their own scope. A language-independent backend assertion need not + be copied nine times. Workflow continuity across that failure still needs the + affected SDK workers. + +## Public experiment inventory + +Server commands below live in `scripts/conformance/` in +[Server](https://github.com/durable-workflow/server/tree/main/scripts/conformance). +Sample App commands and examples live in +[Sample App](https://github.com/durable-workflow/sample-app/tree/main/polyglot). +The scenario manifests define required evidence within their existing scope. +Their SDK installation lists do not widen that scope. + +| Experiment and existing entry point | Executed language scope | Remaining portable behavior to qualify | +| --- | --- | --- | +| Authoring, inputs/results and remote activities: Sample App `scripts/polyglot.sh`, full matrix in `polyglot/README.md`; Server `activities-published-artifacts.sh` | Featured PHP → Python → Rust journey and the full nine-direction activity matrix are separate commands. Server's activity manifest primarily uses embedded PHP/Python activity cells, with a focused PHP SDK heartbeat-renewal shard. | The nine successful result directions do not prove Rust retry, timeout, duplicate completion or activity-attempt recovery. Run these behaviors with actual SDK workers and persisted attempts. | +| Child workflows: Server `child-workflows-published-artifacts.sh`; Sample App `polyglot/child-workflows/` | Server's four embedded PHP/Python parent-child directions and Sample App's nine PHP/Python/Rust SDK directions. | Sample App checks successful child result/history. Add Rust-involving child failures, restart/replay and cancellation. Do not project the Server runner's PHP/Python failure/restart cells onto Rust. | +| Heartbeats: Server `heartbeats-published-artifacts.sh`, `heartbeats-python-published-artifacts.sh`, `heartbeats-rust-published-artifacts.sh`, shared wave runner | All three actual SDK worker heartbeat loops, cadence, metrics, stale transitions and routing exclusion. | Use each runtime's scenario results. A worker heartbeat pass is separate from activity progress heartbeat and cancellation observation. | +| Timers: Server `timers-published-artifacts.sh`; Sample App `scripts/sdk-timers.sh` | Server's embedded PHP timer scenarios. Sample App supplies separate PHP/Python/Rust SDK completion, worker SIGKILL/cold replay, Server restart and cooperative cancellation cells. | Concurrent distinct deadlines and timer-bearing application upgrades need separate SDK execution. | +| Sagas: Server `sagas-published-artifacts.sh`; Sample App `polyglot/sagas/` | Server's PHP/Python workflow-compensation directions; Sample App's five Rust-involving directions, typed compensation failure and restart before compensation. | Process loss during an external compensation side effect, duplicate delivery and external-effect recovery require explicit side-effect fixtures. Restart before compensation does not establish them. | +| Schedules: Server `schedules-published-artifacts.sh`; Sample App `polyglot/schedules/` | PHP/Python creators and workers in Server; PHP and Python creators targeting Rust in Sample App. | The Rust examples check one automatic interval fire. Run cadence, restart, overlap/backfill and failure recovery for those directions. Calendar folds/gaps are temporal cases, not extra creator languages. | +| Signals and queries: Server `signals-queries-published-artifacts.sh` | PHP/Python/Rust clients and workers, including cross-language calls, Rust query immutability and cold-restarted instance state. | Read the direction-specific scenarios and histories. The crate pin alone proves none of them. | +| Workflow updates: Server `workflow-updates-published-artifacts.sh` | Embedded probe, actual PHP and Python SDK worker/client shards, CLI and Waterline diagnostics. | Rust exposes update handlers and clients but has no execution shard in this runner. Add Rust handler/client directions, duplicate request, failure and replacement. Rust refuses synchronous pre-accept validators. Test that refusal explicitly and retain PHP/Python validation cells. | +| Deterministic replay: Server `replay-published-artifacts.sh` | Separate installed PHP, Python and Rust replay shards, command/history matching and codec behavior. | Corpus replay does not prove worker process recovery or migration. Pair affected histories with their runtime restart and upgrade cases. | +| Workflow lifecycle: Server `workflow-lifecycle-host-published-artifacts.sh` | Docker host wrapper builds the exact published Rust probe and starts the published Server; inner runner includes PHP/Python/Rust lifecycle surfaces. | Select real continuation, retry/timeout, cancellation and termination scenarios. Terminal cancellation and cooperative cleanup are separate contracts. | +| Worker versioning: Server `worker-versioning-published-artifacts.sh` | Actual PHP/Python build cohorts and mixed pinning, plus Python no-compatible-worker, replay and adversarial shards. | Rust build pinning, promotion, incompatible refusal, replacement and cold replay need an execution shard. | +| Namespaces: Server `namespaces-published-artifacts.sh` | Server authorization/routing, PHP and Python SDK paths, CLI and Waterline namespace surfaces. | Rust clients/workers need positive and negative namespace execution. A shared Server authorization assertion alone does not prove their header and credential composition. | +| Search attributes: Server `search-attributes-published-artifacts.sh` | PHP SDK, Waterline and PHP/Python codec shards, or supplied host matrix evidence. | Add Rust authoring/client filters and value round trips. Supplied evidence must come from a real worker run, not a hand-written passing result. | +| Nexus: Server `nexus-published-artifacts.sh` | Published PHP/Python caller/service execution, shared-service behavior, replay and cancellation probes. | Require all four supported caller/service directions and their actual async completion, typed failure and cancellation evidence. Rust is outside the current API scope. | +| 1.x → 2.x migration: Server `migration-published-artifacts.sh` | Supported 1.x PHP artifacts and their durable state, target PHP/Python and operator surfaces. | Rust has no 1.x source artifact to migrate. Rust starting against the migrated target is a meaningful post-migration case. SDK upgrades within 2.x/3.x are separate from this engine migration. | +| Compatibility skew: Server `skew-published-artifacts.sh` | CLI, PHP/Python SDK, Waterline and worker protocol pairings. | Add Rust's supported and refused protocol pairings. A major SDK package number is not itself a worker protocol compatibility boundary. | +| Principal attribution: Server `principal-attribution-published-artifacts.sh` | Named/anonymous/raw HTTP actors, PHP/Python clients, worker completion/failure, CLI and Waterline. | Rust SDK credentials and worker history need their own attribution and spoofing-refusal execution. | +| Cooperative cancellation and worker affinity: SDK integration suites, public qualification records and the timer command above | Source integration tests exist in every SDK. Published local-activity/session and mixed-cancellation qualifications are distinct release evidence. | Keep installed-package proof separate from source CI. Check current capabilities: Python/Rust local activities and sessions are released, while sticky execution remains refused and is owned by [.github #124](https://github.com/durable-workflow/.github/issues/124). Older affinity manifest expectations must not override the tested tuple's capabilities. | +| Failover and diagnosis: Server `single-region-failover-published-artifacts.sh`, `agent-operability-published-artifacts.sh`; [operator diagnosis drill](operator-diagnosis.md) | Published container/backend failover and public API/CLI/Waterline operator paths. | Backend recovery success is not three-language workflow continuity. Include affected workers and pending work when that is the customer claim. Embedded Laravel injection/testing remains a PHP framework-specific case. | +| Managed Cloud | Private Cloud repository's isolated qualification | Keep account, provisioning, provider chaos and commercial qualification private. Public SDK protocol coverage does not establish managed-plan capacity or Cloud recovery. | + +## Run and interpret a cell + +1. Freeze the seven-component published tuple and runner revision. Select a + supported capability from those installed artifacts, not a different checkout. +2. Follow the owning runner's `--help` or example README. Use its isolated project, + worker/client credentials and queues. For Sample App timers, resolve the + checked-in tuple and run `scripts/sdk-timers.sh` as documented in + [`polyglot/timers/`](https://github.com/durable-workflow/sample-app/blob/main/polyglot/timers/README.md). +3. Observe the actual language, run identity, result and persisted lifecycle + events. For retries/timeouts record attempts and original deadlines. For + cancellation distinguish request, committed delivery, cleanup and terminal + outcome. For recovery record the physical failure and replacement process. +4. Reject early timer fire, changed deadlines, lost acknowledged work, duplicate + completion, unexecuted directions and drains that were not measured. Keep a + harness failure separate from a product failure. +5. Remove the task stack and scratch state, including after failure. Report the + exact command, UTC interval, tuple, runner SHA and scenario outcomes in the + owning issue. Retain bounded diagnostic output through the existing GitHub + Actions artifact mechanism when needed. Never commit generated run evidence. + +This inventory was checked against Server `d97727113cef0d92228627eecb47988b0955aad9` +and the public Sample App timer change. A new release still needs new runtime +results for its affected cells. This document is not a second release tracker.