diff --git a/docs/0.7-planning-brief.md b/docs/0.7-planning-brief.md new file mode 100644 index 00000000..a87c8f57 --- /dev/null +++ b/docs/0.7-planning-brief.md @@ -0,0 +1,625 @@ +# Awa 0.7 Planning Brief + +> **Purpose.** This document is a consolidated context pack for planning the Awa 0.7 +> development cycle. It captures (1) what Awa is, (2) what the "massive" 0.6 release +> actually shipped, (3) the architectural landscape and open decisions, and (4) a full +> inventory of every open issue — with the authoritative `v0.7.0` GitHub milestone +> distinguished from unmilestoned candidates. It is written to be fed directly to a +> planning model. Facts are sourced from the repo (`README.md`, `CHANGELOG.md`, +> `docs/adr/`, `docs/archive/0.6-storage-design/`) and the live GitHub issue tracker as +> of 2026-07-05. + +--- + +## 1. What Awa is + +**Postgres-native job queue for Rust and Python.** (Māori: *awa* = river.) + +Positioned deliberately *between* two neighbours (`docs/positioning.md`): + +- More capable than a Postgres **event queue** (PgQue / PgQ lineage — event log, consumer + cursors, retention-first). Awa adds priorities, unique jobs, cron, callbacks, DLQ, richer + lifecycle. +- Less ecosystem-bound than a language-specific **job framework** (River/Go, Oban/Elixir). + Awa runs **Rust and Python workers on the same queues, same storage engine**. + +One hard dependency: Postgres. No Redis, no RabbitMQ, no external `pg_cron` ticker — dispatch, +rescue, rotation, prune, and cron are all **leader-elected inside the worker fleet**. + +### Delivery contract (from README) +- **Transactional enqueue** — insert jobs inside the business transaction; commit = visible, + rollback = gone. +- **At-least-once**, not exactly-once. Stale completions rejected; stuck work rescued. + Idempotent handlers recommended. +- **"No lost work under failure" outranks clever fast paths.** +- **Strict FIFO per `(queue, priority)` by default**; operators can opt a hot queue into + **partitioned FIFO** by raising `enqueue_shards` (ADR-025) — an explicit semantic trade + (SQS Standard vs FIFO / Kafka partition count), not a free win. + +### Workspace crates +| Crate | Purpose | +| --- | --- | +| `awa` | Umbrella — re-exports `awa-model` + `awa-worker` | +| `awa-model` | Types, queries, migrations, admin ops | +| `awa-macros` | `#[derive(JobArgs)]` | +| `awa-worker` | Runtime: dispatch, heartbeat, maintenance | +| `awa-ui` | Web UI (axum API + embedded React) | +| `awa-cli` | CLI binary | +| `awa-python` | PyO3 module (`pip install awa-pg`) | +| `awa-testing` | Test helpers (`TestClient`) | +| `awa-seaorm` | New in 0.6 — transactional enqueue alongside SeaORM writes | +| `awa-metrics` | New in 0.6 — shared OTel counters for non-runtime callers | + +### Correctness posture +Core invariants are checked by **TLA+ models** (`correctness/`): no duplicate processing after +rescue, stale completions rejected, no claim/rotate/prune deadlock, DLQ round-trip safety, +prune-segment emptiness, heartbeat-driven short-job rescue. Two model families: +- **Segmented-storage family** (`AwaSegmentedStorage`, `…Races`, `AwaStorageLockOrder`, + `AwaSegmentedStorageTrace`) → ADR-019/020. The trace model replays concrete runtime-test + event sequences against the spec. +- **Worker-runtime family** (`AwaCore`, `AwaExtended`, `AwaBatcher`, `AwaCbk`, + `AwaDispatchClaim`, `AwaViewTrigger`, `AwaCron`). + +--- + +## 2. The 0.6 release — what shipped + +**0.6.0 tagged 2026-07-04.** The headline: the append-only **queue-storage engine** becomes +the **default and supported substrate**, replacing the row-mutating "canonical" engine. +Migrations v001–v039. The whole release was gated on a single benchmark result (#169, below). + +> "The 0.6 line makes the append-only **queue-storage engine** the default … The +> [#169](https://github.com/hardbyte/awa/issues/169) pinned-MVCC dead-tuple gate **passed** +> on `main`: after the rc line removed the last dominant hot-row update path (the +> `queue_claim_heads` ready-segment routing cache), a 60-minute idle-in-transaction soak +> holds bounded queue depth through the pinned hour and drains fully in recovery." +> — `CHANGELOG.md` + +### 2.1 Storage & durability (the core of the release) +The through-line was **eliminating every hot-row UPDATE on the claim/complete loop** so dead +tuples stay bounded under a pinned MVCC horizon: + +- **Queue-storage engine, default-on** (ADR-019). Append-only `ready_entries`, + `deferred_jobs`, `done_entries`, `dlq_entries` + partitioned receipt ring. +- Lane cursors moved from MVCC-updated rows to **Postgres sequences** (#321, v027). +- Completion/DLQ/retry/discard no longer `DELETE FROM ready_entries` — append-only ready + segments + a **tombstone ledger** (#323, v028). +- Terminal counts append signed rows to a **delta ledger** folded asynchronously by the + maintenance leader (#329, v030, ADR-026). +- **The last dominant dead-tuple accumulator is gone**: the per-claim `queue_claim_heads` + routing-cache update was removed; claim now routes through the `ready_segments` control + plane (#355, v039). This was the final change that let the #169 gate pass. +- Rollup mutation and segment truncation **stand down while another backend pins the MVCC + horizon**, then catch up once it clears (#333). +- **Compact receipt claim/completion batches** (#352, v036–v038, ADR-026) — receipt-backed + successful completions write compact terminal batches instead of one `done_entries` row per + job. `{schema}.terminal_jobs` is the public terminal read surface; **direct SQL against + `done_entries` must not assume all completed jobs live there.** +- **Failed-terminal retention floor** (#337, v032, ADR-032) — non-DLQ `failed` rows stay + retryable ≥ `failed_retention` (default `72h`); aged-out rows fold into a monotonic + `QueueCounts.pruned_failed` so loss is visible. +- **Receipt-plane ring partitioning** (ADR-023) — `lease_claims` / `lease_claim_closures` + partitioned by claim slot, rotated by maintenance (#349, v033–v035). +- **Per-claim deadlines** in receipts mode (`QueueConfig.deadline_duration`). +- **Staged 0.5→0.6 transition tooling**: `awa storage prepare` / + `prepare-queue-storage-schema` / `enter-mixed-transition` / `finalize` (`--wait`/`--check`) + / `abort`, plus a storage-transition readiness UI (#298, #299). `awa migrate` is now the + **single owner of `awa.*` DDL** (#308, v023); fresh installs auto-finalize. + +**The staged transition is a one-way door.** `abort` works up to `mixed_transition` *before any +queue-storage row is written*; once queue-storage work is accepted, the only rollback is a +database restore (0.5 workers can't claim queue-storage work). See +`docs/upgrade-0.5-to-0.6.md`. + +### 2.2 Operator surfaces +- **Dead Letter Queue** (ADR-020) — per-queue `dlq_enabled`; `awa dlq depth|list|retry| + retry-bulk|move|purge`, UI tab, Rust/Python APIs. +- **Durable batch operations** (#328, v029, ADR-030) — crash-safe, maintenance-driven bulk + `set_priority` / `move_queue` with preview/submit/list/get/cancel/purge (HTTP, CLI, UI); + Python parity (#332). This is the first-class reprioritization surface (#307). +- **Transaction-scoped admin cancel** (#357/#358) — `admin::cancel_tx` / + `cancel_by_unique_key_tx` take a caller `&mut PgConnection` and run cancellation + the + cooperative `awa:cancel` NOTIFY atomically with the caller's work. +- **Descriptor catalog** (ADR-022) — code-declared queue/job-kind metadata drives UI labels + and drift detection. +- **Cron missed-fire policy** (`coalesce` default / `catch_up`) + **pause/resume** (#320, + v026). +- **`awa-pg[ui]` extra** + `python -m awa serve`. +- **Managed-Postgres deployment guide** (`docs/deploying-on-managed-postgres.md`) — sizing, + Cloud SQL IAM (`cloudsqlsuperuser`), AlloyDB notes, auth-proxy sidecar, MVCC guidance. + +### 2.3 Producer APIs & tuning +- **`PartitionedQueue`** (#348, ADR-031) — deterministic routing of one hot logical queue over + N physical partitions. Partition 0 *is* the logical name (`email`, `email__p1`, …) so direct + enqueues stay consumable; domain-separated key hash composes with ADR-025 enqueue-shards. + **Replaces the beta-series `QueueFanout` (breaking rename).** +- **Per-job routing in Python COPY batch APIs** (#348) — `insert_many_copy` / + `enqueue_many_copy` accept `opts=[...]` overriding `queue` / `ordering_key` per job. +- **Transactional follow-up jobs** (ADR-029, #285/#288) — `ClientBuilder::on_completed_enqueue` + registers a follow-up job inserted in the same tx as the lifecycle transition. *This is the + durable-side-effect primitive that ADRs 027/028 lean on.* +- **Callback-only router / user-owned callback layers** (#291/#293) with configurable URL + prefix; `WaitingForCallback` lifecycle event + `resolve_callback` / `complete_external` / + `fail_external` / `retry_external` (#276). +- **Direct queue-storage COPY producer path** — Rust `QueueStorage::enqueue_params_copy`, + Python `enqueue_many_copy` — as the documented high-volume entry point. `insert_many_copy` + is the compat route, **not** the fast path. +- **`InsertOpts::ordering_key`** + `awa.queue_meta.enqueue_shards` (ADR-025) — per-queue + semantic switch; default `1` = strict FIFO, `≥2` = partitioned FIFO. +- **Handler futures no longer require `Sync`** (#331) — only `Send + 'static`. + +### 2.4 Telemetry +- **`awa-metrics` crate** so `awa-ui` / `awa-cli` emit the same OTel counters as the worker. +- Sub-second buckets on `awa.job.wait_duration`; `awa.enqueue.batch_size` / + `awa.enqueue.duration` histograms; `awa.enqueue.shard` attribute on `awa.job.claimed`. +- **`queue_counts_fast`** (#289) — index-only depth probe (under-counts unrolled terminal + rows; use `queue_counts` for exact). +- **Exact terminal counts without `done_entries` scans** (#290 via #304/#306) — folded + counters serve exact reads once the trust marker is set + (`awa storage rebuild-terminal-counters`). +- Maintenance branch duration histograms + `awa_maintenance_branch_overrun_total{branch}` + (#302). + +### 2.5 Breaking changes (0.6 beta/rc series — matter for early adopters) +- `QueueFanout` → `PartitionedQueue` (no alias): `width`→`partitions`, `queue_fanout`→ + `partitioned_queue`, Python `*_per_queue`→`*_per_partition`; **routing differs from beta.2** + (partition 0 = logical name, domain-separated hash) so a key may land on a different + partition — drain fanout queues before upgrading. +- `QueueCounts.completed` → `QueueCounts.terminal` (now counts completed+cancelled+discarded); + gains `pruned_failed`. +- `retry_failed_by_kind`/`by_queue` return `RetryFailedOutcome { retried, matched, + pruned_failed_count }` instead of `Vec`. +- `JobEvent`/`UntypedJobEvent` gain a `WaitingForCallback` variant (exhaustive matches need a + new arm). +- `PruneOutcome::Pruned` gains `carried_failed_rows`; `QueueStorage::prune_oldest` takes + `failed_retention: Duration`. + +### 2.6 Benchmark reality (#169) — the honest ceiling +This is the single most important context for 0.7. **0.6 did not make Awa immune to long +readers; it made the degradation bounded and recoverable, and documented the mitigation.** + +- Harness: 1 replica, 64 workers, 800 jobs/s offered, 60-min idle-in-transaction reader + (`hardbyte/postgresql-job-queue-benchmarking`). +- Clean phase holds the offered rate at shallow median depth. **Under the pinned reader, + completion sags but depth stays bounded (no dead-tuple cliff) and drains fully once the pin + releases** — append-only segments hold zero dead tuples; residual pressure confined to + terminal-counter and ring-state metadata. This shape *is* the #169 gate and it passed. +- **Every per-row-state-machine Postgres queue (River, Oban, pg-boss, Graphile, pgmq) shares + this degradation shape.** pgque avoids it only by dropping features (no per-row retries, + heartbeats, cancellation). +- Documented mitigation ("MVCC discipline"): give Awa its own database if the app runs long + `REPEATABLE READ`/`SERIALIZABLE` transactions; bound sessions with + `idle_in_transaction_session_timeout`; alert on `pg_stat_activity.xact_start` age. +- Separate result: Cloud SQL PG18 16-vCPU sweep (v021 shard-aware lane indexes) moved + end-to-end drain on a 3.5M-row backlog from ~1,300 → ~8,800–10,600 jobs/s. + +Headline reference numbers (README): **9.5k jobs/s, 22 ms p95 pickup, 417 exact final dead +tuples** on a 5k-job runtime soak; enqueue ~30k/s single-producer, ~100k/s multi-producer. + +--- + +## 3. Architectural landscape & open decisions + +### 3.1 ADR status snapshot (`docs/adr/README.md`) +- **Accepted & central to 0.6:** 019 (queue storage engine, supersedes 012), 020 (DLQ), 023 + (receipt-plane ring), 025 (sharded enqueue heads), 026 (narrow terminal history), 029 + (transactional follow-up jobs), 030 (batch ops), 031 (partitioned queues), 032 (failed + retention floor). +- **Proposed (not yet built — these are the forward-looking deployment-shape ADRs):** + - **ADR-027 — Callback ingress as a deployable surface.** Split the signed callback receiver + away from the admin UI/API so it can be public/partner-facing while the admin UI stays + private. Introduces a surface taxonomy: Admin UI/API (private) · Callback receiver + (public, signed) · Worker runtime (internal) · Maintenance runtime (internal). + - **ADR-028 — Maintenance-only runtime role.** A persistent role that runs promotion, + rescue, prune, and metadata maintenance **without claiming or executing user jobs** — + required so callback-only / serverless deployments don't leave time-based state frozen. +- **Rejected (historical):** 024 (deferred `done_entries` materialisation). + +### 3.2 Cross-cutting constraints any 0.7 proposal must respect +- **Postgres-only (ADR-001).** No extensions Awa doesn't already require. PG14+ floor, PG18 + supported. +- **No new hot mutable table under a pinned MVCC horizon.** Any new per-key / per-tenant / + per-dependency state family must use the same **delta-ledger discipline as ADR-026** or it + reintroduces the #169 problem. +- **ADR-029 transactional follow-up semantics must compose** with anything new. +- **Non-goal: workflow engine.** The PRD explicitly lists "workflow engine" as a non-goal — no + DAG authoring, no saga orchestration. This directly bounds #14 and #81. +- **Transactional-enqueue tension (ADR-006):** rejecting an enqueue inside a user transaction + turns a queue-health condition into a business-transaction failure — relevant to backpressure + (#341). + +--- + +## 4. Open issue inventory (20 open) + +**Authoritative signal: 8 issues carry the GitHub `v0.7.0` milestone.** These are the +committed/queued 0.7 scope. The remaining 12 are unmilestoned candidates. Author is `hardbyte` +(project owner) except #256 (external) and #346 (bot). + +### 4.1 — In the `v0.7.0` milestone (committed scope) + +| # | Title | Labels | Notes | +| --- | --- | --- | --- | +| **295** | RFC: v0.7 storage engine — segment storage + cursor allocator | feature, correctness | **The defining 0.7 item.** RFC-stage, no code. | +| **246** *(not milestoned — see 4.2, but tightly bound to 295)* | | | | +| **360** | Test harness: run suite under both storage engines | — | Quality/safety prerequisite for evolving storage. | +| **303** | maintenance: split rescue+promote into separate tokio task (v0.7) | operational | **Conditional** — telemetry-gated, may close wontfix. | +| **282** | Add a maintenance-only runtime role | feature, operational | Implements ADR-028. | +| **143** | Split awa-api from awa-ui | — | Decouple API response types from embedded React. | +| **118** | Serverless-friendly dispatch: `tick()` endpoint | — | Zero-infra deployments; pairs with ADR-028. | +| **110** | Tracing (end-to-end distributed tracing) | feature | Context propagation enqueue→exec, across PyO3. | +| **14** | Job dependencies: run B after A completes | feature | Must stay minimal (A→B, not DAGs) per non-goal. | + +#### #295 — RFC: v0.7 storage engine — segment storage + cursor allocator *(flagship)* +Milestone `v0.7.0`. **Design RFC, no implementation.** Directly targets the #169 residual +degradation (799 → 387 jobs/s at 800 offered/s over a 2h pinned phase) — the class of behaviour +0.6's ADR-012/023/025/026 mitigated but did not eliminate. + +> "the lifecycle table itself is the wrong shape for sustained, long-reader-exposed +> workloads." … "the storage *shape* is right but the *claim allocator* is the load-bearing +> piece — the design has to start there, not at the segment format." + +Direction: **append-only rotation segments** for job lifecycle + a **tiny per-segment claim +ledger** (the only mutable hot rows) + a **cursor allocator** that moves readers forward across +segments instead of anti-joining over prunable history. Five explicit RFC questions: +1. **Claim allocator design** — simplest allocator giving SKIP-LOCKED-equivalent fairness that + supports per-row retries/heartbeats/cancellation and does *not* degrade under a pinned + horizon. Single advisory-locked cursor table? Per-shard hash-routed cursor? Something else? +2. **Receipt-plane integration** — how ADR-023 ring partitioning + ADR-026 narrow terminal + history compose with rotation segments. +3. **Migration story** — 0.6→0.7 is a *second* storage migration right after 0.5→0.6. Staged + (like 0.5→0.6) or hard cutoff? +4. **Comparable designs** — FoundationDB record layer, ScyllaDB commit log, pgque. *What stops + pgque from supporting per-row retries/heartbeats?* +5. **TLA+ coverage** — what must be re-modelled before it ships. + +Constraints: Postgres-only, PG14+/PG18, API-compatible with 0.6 where feasible (operator-visible +migration, **not** handler-visible), ADR-029 semantics must compose. Contribution path: draft +ADR (next free number) once a direction has consensus. + +#### #360 — Dual-engine test harness +The caller-facing contract is meant to be **engine-invariant** (canonical vs queue_storage) but +the default integration harness forces `canonical`, so queue_storage implementations are only +covered by a dedicated suite. + +> "This blind spot is how the queue-storage `cancel_by_unique_key` candidate query +> (`{schema}.leases.unique_key`, a column that does not exist on `leases`) shipped … the +> queue-storage SQL branch was dead code to CI." *(fixed #359 / 0.6.0-rc.4)* + +Experiment: running `integration_test` under `AWA_TEST_ENGINE=queue_storage` → **31 pass / 13 +fail, all 13 test artifacts** (canonical-only raw-SQL helpers, admin-metadata cache assertions, +test-isolation leakage), **zero real engine bugs**. Proposes engine-aware harness helpers, an +engine guard for canonical-only tests, a CI matrix dimension, and a **forward-compat test** (old +binary, e.g. awa 0.5.7, against a freshly-V39 DB — currently untested). A spike branch with the +parameterization + triage exists locally. *High leverage: this is the safety net for shipping +#295.* + +#### #303 — Split rescue+promote into a separate tokio task *(conditional)* +Follow-up to #242 (closed via #302). 0.6 shipped only *observability* +(`awa.maintenance.branch.duration/overrun`); this issue **only activates if fleet telemetry +shows non-trivial overrun on user-visible branches**, else closes wontfix. +> "Conditional follow-up — open for visibility, not a foregone conclusion." TLA+ needs no +> re-modeling (every branch is still its own SQL tx). + +#### #282 — Maintenance-only runtime role (implements ADR-028) +`.maintenance_only()` library API + `awa maintenance run` CLI. Acceptance: does NOT claim jobs / +invoke handlers / dispatch HttpWorker, but DOES run promote/rescue/rotation/prune/cleanup/ +metadata and preserves leader-election + shutdown. "Composes with #242 rather than baking in +another monolithic maintenance loop." Unblocks callback-only (#118, ADR-027) deployments. + +#### #143 — Split awa-api from awa-ui +`awa-ui` mixes REST API (axum handlers, response types) with the embedded React frontend +(rust-embed). +> "The API currently returns `JobRow` (a database row struct deriving `FromRow`) directly to +> HTTP clients. This leaks the DAL layer into the API contract." +A `JobResponse` wrapper was added as a first step; proposes a dedicated `awa-api` crate so +`awa-cli` can depend on it without the React build. *Note interaction with #343 UI auth and #344 +Helm — all touch the serve surface.* + +#### #118 — Serverless `tick()` dispatch endpoint +HttpWorker (ADR-018) made *execution* serverless, but dispatch/maintenance still needs an +always-on process — overkill for hobby/Supabase-scale apps. +> "The system is frozen until the next request arrives." (the no-traffic maintenance gap) +Proposes a **bounded** `tick()` (each step `LIMIT 100` for predictable duration) + `POST +/api/tick` driven by a cheap external scheduler (Cloud Scheduler, Vercel cron, GH Actions). +Open Qs: Client method vs standalone; auth by default; leader election ("Probably: skip it, let +SKIP LOCKED handle contention"). *Pairs directly with ADR-028 / #282.* + +#### #110 — End-to-end distributed tracing +Awa emits OTel spans/metrics but tracing is **component-local** — parent context doesn't +propagate enqueue→execution, especially across the Rust↔Python FFI (ADR-004) and transactional +enqueues. Proposes storing serialized `trace_context` (JSONB/binary, <200 bytes) on the job, +injecting as parent at claim/dispatch, propagating across PyO3, recording span status on +completion (guarded by `run_lease`), extending to callbacks/cron/maintenance. W3C tracecontext +default; ship Rust→execution first, then the Python boundary. + +#### #14 — Job dependencies (A→B) +> "after the PDF is generated, send the email" — currently requires manual next-job insert. +Three options: `depends_on` column (maintenance promotes B when deps complete); a completion +callback (`InsertOpts { on_complete }`, single-step); or document the manual approach. +> "The PRD explicitly lists 'workflow engine' as a non-goal. This should stay minimal — simple +> A→B chains, not DAGs." Tightly related to #81. + +### 4.2 — Unmilestoned candidates (12) + +**Storage / correctness / perf** +- **#246 — Claim hot-path slows ~33% with deadline rescue enabled at high concurrency.** + *(4 comments; the most-characterized bug in the tracker.)* At 1×256 near saturation, + per-claim deadline rescue costs ~33% throughput / ~50% p99, spread uniformly across the claim + path (not one query) → shared-resource contention. Measured: completion 5,419→3,649 jobs/s + (−33%), e2e p99 100→151 ms (+51%). Primary hypothesis: + > "`awa.lease_claims` working set inflates when rescue is on, and every other query that + > touches `lease_claims` … pays the cost in index lookup time / page fetches / lock + > contention." + Safety constraint: **do not disable `deadline_at` writes to fix it** — per-claim deadline + rescue is the documented stuck-worker fallback the chaos suite relies on. A partial-index fix + needs a migration plan (`lease_claims` is partitioned). Reproduces on 0.6.0-alpha.6/7; + `pg_stat_statements` snapshots committed. *Strongest concrete correctness/perf bug for 0.7; + bound to the #295 receipt-plane redesign.* + +**Dispatch / scheduling features** +- **#347 — Design first-class partitioned queues (ADR-031 draft).** Mostly shipped in 0.6 + (`PartitionedQueue`), but the issue documents the **correlated-hashing routing defect** the + 0.6 rename fixed: `queue_for_key` and the storage enqueue-shard pick used *the same 64-bit + hash under two moduli*, so at `width=4, enqueue_shards=4` each partition's keyed traffic + landed on exactly one shard — "ADR-025 is silently negated for keyed workloads." Fix = + domain-separate the partition hash. Largely closed by 0.6; verify residual open questions + (weighted mode, exact naming) are resolved or move to done. +- **#340 — Per-key execution control: concurrency limits & fairness within a queue.** The main + multi-tenant dispatch gap (cf. Oban Pro partitioned concurrency, BullMQ groups). Two gaps: + per-key concurrency ("≤N in-flight per tenant") and cross-key fairness (one hot tenant + starves others until priority aging). Key constraint: + > "per-key state must not become a new hot mutable table under pinned MVCC horizons (#169) — + > any per-key counter family needs the same delta-ledger discipline as ADR-026." + Enforcement point undecided: claim-time skip (interacts with cursor monotonicity/tombstones) + vs post-claim gating (burns claim throughput). Explicitly *not* workflow semantics. +- **#341 — Backpressure: queue-depth limits & producer-side flow control.** Nothing stops + producers outrunning completion; the COPY path (ADR-008) makes it trivial. Cites #246 numbers + (9,982/s enqueue vs 4,016/s completion, depth peaked 1.26M, 33s e2e p99). Building blocks + exist (`queue_counts_fast` #289, lane-head cursors #330). + > "A soft-signal default with opt-in hard rejection is probably the right shape." … "a + > depth-aware batch producer … would capture most of the value without changing enqueue + > semantics at all." + +**Operational / deployment** +- **#344 — First-party Helm chart (or kustomize base).** No installable k8s artifact today. + Cover Worker Deployment (+auth-proxy sidecar), Migrations Job (`awa migrate` as single DDL + owner), UI Deployment/Service (`--read-only`), probes. Must state explicitly that + maintenance/cron are **leader-elected inside workers (ADR-007), not a k8s CronJob.** Possible + sub-task: workers may lack a health endpoint for probes. +- **#343 — Admin UI authentication.** Today only `--read-only` + "put it behind your ingress", + no worked example, no built-in auth — despite the UI exposing retry, cancel, DLQ purge, batch + ops, cron pause, and **storage-transition controls** (mutating, operator-grade). Proposes + (smallest-first): document an oauth2-proxy/OIDC pattern; add optional `AWA_ADMIN_TOKEN` + static-token/basic-auth ("not an SSO replacement"); defer authz tiers. *Pairs with #143 and + ADR-027 surface split.* +- **#335 — CI: Rust tests job takes 26+ min; shard it.** PR-gating "Rust tests" took 26m22s on + a docs PR, mostly serialized waiting on real-time windows (one binary 180s). + > "A test that needs >60s of wall clock is almost always waiting on a real-time window … These + > windows are configuration, not contract — the test should pin the behavior, not the + > production default duration." + Proposes CI matrix sharding (per-shard Postgres), cargo-nextest, sub-second fixture windows. + Target: worst shard <8 min. *High developer-velocity leverage across the whole 0.7 cycle.* +- **#346 — Nightly chaos/benchmark failure (2026-06-11).** Bot-filed; only `rust-nightly` + failed (tla-storage / postgres-failover-smoke / python-nightly all passed). *Triage/close.* + +**API surface / architecture** +- **#342 — Stabilize the SQL enqueue contract as a public producer API.** Common shape: a third + language (Node/Go/JVM) on the *producer* side only. `awa.insert_job_compat(...)` is already a + complete SQL entry point (#314 documented SQL-only install/upgrade). + > "What's missing is the **decision and contract**, not the mechanism." … "`unique_key` is a + > BLAKE3 hash computed client-side (ADR-002). A non-Rust producer needs the exact derivation … + > documented, or uniqueness silently diverges between languages." + Out of scope: official client libraries for other languages. *Mostly a decision + docs + + contract-tests effort; positioning win.* +- **#143** *(also in milestone — see 4.1)*. +- **#256 — Optional SeaORM integration crate.** The one external-contributor issue + (`xyzbety`); a thin `awa-seaorm` crate surfacing `sqlx::PgPool` from + `sea_orm::DatabaseConnection`. **Note: `awa-seaorm` already exists in the workspace and its + publish was added to the crates.io chain (`da69728`) — likely already landed; verify and + close.** + +**Research / positioning (no committed direction)** +- **#81 — Research first-class orchestration within Awa's scope.** Evaluate `send`/`call`/ + `delay`-style multi-step APIs against the non-goal. + > PRD: "Postgres-native job execution for Rust and Python. One queue engine, two first-class + > languages." Any proposal must restate the non-goals and explain why it isn't a workflow + > engine. Related to #14. +- **#83 — Research official ingress adapters** (webhook-to-job, Kafka event-stream, HTTP + ingress). Deliverable: keep ingress app-defined / ship reference adapters only / add + first-party packages — identify the smallest useful first step. + +--- + +## 5. Synthesis — how the 0.7 scope hangs together + +There are **three coherent workstreams** plus a bug and some research: + +**A. The storage endgame (the reason 0.7 exists).** +`#295` (segment-storage + cursor allocator RFC) is the flagship, and it is still *pre-design*. +`#360` (dual-engine harness) is its safety net and should land first — it is bounded (spike +branch exists) and closes the exact defect class that a storage rewrite would otherwise risk. +`#246` (deadline-rescue contention on `lease_claims`) is empirically the same receipt-plane +surface #295 must redesign, so its resolution and the RFC should be reasoned about together. +`#303` is a telemetry-gated maybe. +- *Biggest schedule risk:* #295 is an RFC with five open design questions and implies a + **second storage migration in consecutive releases** (0.5→0.6→0.7) — migration UX and + operator trust are as much the work as the engine. + +**B. Deployment-shape maturation (ADR-027 + ADR-028 becoming real).** +`#282` (maintenance-only role, ADR-028) + `#118` (serverless `tick()`) + the ADR-027 callback +ingress split together enable "callback-only / no always-on worker" deployments. `#344` (Helm), +`#343` (UI auth), and `#143` (awa-api split) are the operational/packaging complements that make +those deployments real and safe. These are largely independent of A and could ship +incrementally. + +**C. Observability & multi-tenancy features.** +`#110` (distributed tracing) is milestoned and self-contained (storage-column + propagation). +`#340` (per-key concurrency/fairness) and `#341` (backpressure) are the highest-value *new* +capabilities but are at the design-question stage and both carry the #169 "no new hot mutable +table" constraint — they may be better sequenced *after* the #295 allocator lands, since a +cursor allocator changes what per-key state is cheap. + +**D. Bounded feature: #14 job dependencies** — deliberately minimal (A→B), guarded by the +workflow-engine non-goal; #81/#83 are research spikes that gate whether B/C grow further. + +**E. Hygiene:** #335 (CI sharding — velocity multiplier for the whole cycle), #346 (nightly +triage), #256 (likely already done — verify/close), #347 (largely shipped — reconcile/close). + +### Suggested planning questions for the powerful model +1. **Sequencing:** Does #360 (dual-engine harness) block the start of #295 implementation? (It + should — it's the regression net for a storage rewrite.) +2. **#246 vs #295:** Fix the deadline-rescue `lease_claims` contention *within the current + receipt plane*, or fold it into the #295 allocator redesign? (They touch the same surface.) +3. **Migration UX:** A second storage migration in back-to-back releases — staged + (`prepare→mixed→finalize`, like 0.6) or hard cutoff? What's the operator-trust cost, and can + 0.6→0.7 reuse the transition tooling from #298/#299? +4. **Per-key state (#340/#341):** How to add per-tenant concurrency/fairness/backpressure + counters without a new hot mutable table under a pinned horizon — delta-ledger (ADR-026) or + cursor-allocator-native? Is this better *after* #295? +5. **Deployment stream independence:** Can B (#282/#118/#344/#343/#143 + ADR-027/028) ship on + its own cadence, decoupled from the storage rewrite risk? +6. **Non-goal boundary:** #14 + #81 — where exactly is the line that keeps Awa a job queue and + not a workflow engine, and does per-key control (#340) or job dependencies (#14) cross it? + +--- + +## 6. The #169 → 0.6 hot-row chronology (the most important input for #295) + +The closed #169 thread is the empirical spine of the entire 0.6 storage story, and it is the +strongest argument for *why* #295 needs to be an architectural redesign rather than more +symptom-fixing. The pattern is explicit in the owner's own words: **"fix one hot row, the next +one surfaces."** Each dead-tuple source was removed, the pinned-horizon benchmark re-run, and a +new dominant mutable hot row appeared — until the last one turned out to be a query-plan bug in +disguise. + +### 6.1 The original failure (2026-05-17, long-horizon Awa vs pgque) +Shape: 1 replica, 32 workers, **800 offered jobs/s**, phases warmup→clean→60m idle-in-tx→2h +recovery→2h idle-in-tx. + +| System | clean | idle-in-tx 1h | recovery 2h | idle-in-tx 2h | WAL/job | +| --- | --: | --: | --: | --: | --: | +| **Awa** | 799/s | **581/s** (depth 128k) | 799/s (drained ~27 min after pin released) | **387/s** (depth 692k, peak **2.49M**) | ~2.2–2.9 KiB | +| **pgque** | 799/s | 799/s (depth 0) | 799/s | 799/s (depth 0) | **~452 B** | + +Owner's read: *"This is not a correctness failure: no lost jobs or safety violation … It is a +production-operational limit."* And the key comparative fact for #295: **pgque holds the rate +with ~5× less WAL/job**, at the cost of a 1.3–1.4 GB active event table and the weaker event-bus +contract (no per-row retries/heartbeats/cancellation). + +### 6.2 The hot-row lineage (each fix exposed the next) +This ordered list *is* the map of where mutable hot state lived in the per-row model — directly +useful for designing the #295 claim allocator, which must avoid re-introducing any of them: + +1. **`queue_lanes.available_count`** cache — removed v016 / #251 (derive from + `next_seq − claim_seq`). Was ~40k dead tuples in a 4-min window, ~2,500× over live rows. +2. **`queue_claimer_leases` / `leases` / `lease_claims`** — fillfactor + index-shape passes + (#315, `d21e5db`) restored HOT-updates. +3. **`leases.heartbeat_at`** write in receipts mode — removed (#317, "B1"); dropped + `idx_state_hb`, compat reads via `COALESCE(…, attempt_state.heartbeat_at)`. Lease/attempt + heartbeat churn stopped being dominant. +4. **Maintenance noise** — prune backoff + branch hysteresis (#316), duration-margin cooldown + + 250 ms lease-rotate default (#318). Delayed the cliff ~40–50 min but did **not** close it; + `rotate_lease` warning storms persisted around the 50 ms threshold. +5. **Lane cursors** — sequence-backed, off mutable lane rows (#321, v027). +6. **Ready segments** — append-only + tombstone ledger; completion/DLQ/retry/discard stop + `DELETE FROM ready_entries` (#323, v028). *After this, `ready_entries_*`/`done_entries_*`/ + `ready_tombstones_*` held **zero** dead tuples through the pin — permanently.* +7. **`queue_terminal_live_counts`** then became the dominant accumulator: **2.52M dead tuples, + 197 MiB, 85% dead** (post-#323 run). Fixed by the ADR-026 **terminal-count delta ledger + + async rollup** (#329/#330/#333) and **256-way striping** → 0 dead tuples at end of pin. +8. **`queue_claim_heads`** was the last dominant hot row (**130k dead tuples, 80.5% share**, + one live row taking ~40 dead versions/s from the per-claim `ready_segment_*` routing-cache + UPDATE). **#355 removed the cache entirely** — and the EXPLAIN study found *"the cache was + hiding a query-plan bug, not earning its keep"*: the cache-miss scan heapsorted the whole + growing tail; ordering `ready_segments` by `next_lane_seq ASC LIMIT 1` lets the index + short-circuit. Removing it made claim **faster** under the pin (claim p99 88→55 ms, total + idle dead tuples −81%, idle depth max 14,265→450). + +Also surfaced during the hunt: a **dispatcher `JoinSet` leak (~3.2 GB/h/replica)**, fixed +`c0df1df`. + +### 6.3 The gate that finally passed (2026-06-18/24, → 0.6.0-rc.1) +After #355, the corrected shape (1 replica, 32 workers, 800/s, clean 20m → **idle-in-tx 60m** → +recovery 10m) **held 798/s completion through the full 60-minute pin** with bounded depth (max +~14k, mostly ~20) and **full dead-tuple reclaim** on release. #169 closed 2026-06-24. The stable +bar had been *raised* mid-cycle (2026-06-10): from "measurable improvement + honest docs" to +"must **pass** — bounded depth through the pin, drain to ~0 in recovery, **and a named +mechanism** for any residual throughput drop." + +### 6.4 What this hands #295 +- **The claim allocator is the load-bearing piece** — 0.6 already proved the *storage shape* + (append-only segments = 0 dead tuples) works; every remaining problem was in mutable + *control-plane* rows (lane counters, terminal counters, routing cache) and in *query plans*. +- **A working list of hot rows to design out**: lane cursors, terminal counters, claim-head + routing — all now solved by sequences / delta-ledgers / control-plane scans, but a clean + allocator should make them structural, not a series of patches. +- **The WAL gap is the real target**: pgque's ~452 B/job vs Awa's ~2.2–2.9 KiB/job is the + quantified headroom, and RFC question #4 ("what stops pgque from supporting per-row + retries/heartbeats?") is asking exactly how much of that gap is intrinsic to the feature set. +- **A raised, concrete acceptance bar already exists** for judging #295: pass the 60-min pinned + long-horizon shape with a *mechanistic* explanation, not just better numbers. +- **Single-shard 10k/s worst case is still unsolved** (9,982/s enqueue vs 4,016/s complete, + depth →1.26M) — this is the #341 backpressure motivation and a second bench shape #295 should + be measured against, distinct from the MVCC-pin shape. + +## 7. Release-readiness process (from the #197 tracker) — the model for a 0.7 tracker + +#197 is worth studying as the *template* for how 0.7 (esp. #295) will be driven to a tag. It ran +as a live gate checklist across six dimensions, each with explicit pass criteria: + +- **TLA+ / correctness gate** — every storage change re-modelled before merge; trace witnesses + added for receipt rescue, running cancel, receipt-only cancel, callback wait/resume, DLQ + retry, DLQ purge; a P0 blocker was the *mixed-transition executor-gate mismatch* + (`AwaStorageTransitionCurrentGate.cfg` could permit mixed transition without proving live + queue-storage execution capacity). **#295 explicitly asks which models need re-doing.** +- **Transition / operational gate** — the staged `canonical→prepared→mixed→active` machinery, + fresh-install auto-finalize, rolling rehearsal in nightlies, the #180 decisions (finalize + stays explicit + `--wait`/`--check`; **no reverse migrator, no `--require-recent-backup`, + `canonical→queue_storage` documented as one-way like Oban/River/Graphile**; no new audit + table — client-accumulated history only). +- **Nightly / stability gate** — chaos assertions converted from log-scraping (`wait_for_line`) + to DB-state observation (`wait_for_kind_state_count`); the constrained-100× acceptance dropped + once flakes cleared. +- **Performance / MVCC gate** — the whole §6 chronology; the bar was *raised* mid-cycle. +- **Documentation gate** — ADR-019 annotated `**historical:**` where ADR-023 supersedes it; + cross-doc consistency sweep; benchmark docs de-claimed as point-in-time not universal. +- **Release mechanics** — crates published in dependency order (note `awa-seaorm` added to the + chain), wheels (`awa-pg` + `awa-cli`), GitHub release, Docker. + +Process decisions that carry into 0.7: +- **Beta-first.** 0.6 cut `beta.1` while three judgment-call gates (#169, #180, #167/#242) were + still open, to "resolve under beta exposure rather than against a hypothetical population." +- **Milestone reshuffling is explicit.** On 2026-05-12 the owner moved #118, #242, #143 to + v0.7.0 and **refocused #110** ("0.6 got opt-in OTel span context propagation + docs; full + schema / automatic propagation deferred") — so #110's *full* scope is genuinely 0.7 work, and + #303 was spun off from #242 as the conditional v0.7 follow-up. +- **#295 was partly pre-paid.** Its highest-value parts (cursor allocation + append-only ready + segments) were *pulled forward into 0.6* as #321/#323; what remains for 0.7 is the full + lifecycle-table replacement and the non-naive claim allocator. +- **#307 (reprioritization)** closed via ADR-030 durable batch ops (#328/#332), not bespoke + endpoints — a precedent for preferring a general durable-mutation surface over point APIs + (relevant to #340/#341). + +## 8. Key file & reference index +- `README.md` — product overview, delivery contract, benchmarks, examples. +- `CHANGELOG.md` — the full 0.6.0 entry (source for §2) + granular alpha/beta/rc log. +- `docs/positioning.md` — category map, what to rely on. +- `docs/upgrade-0.5-to-0.6.md` — staged transition, rollback boundaries, v017 sharding notes. +- `docs/adr/README.md` — ADR index with status (027/028 = Proposed). +- `docs/adr/027-callback-ingress-surface.md`, `…/028-maintenance-only-runtime-role.md` — the + forward-looking deployment ADRs. +- `docs/adr/019/023/025/026/029/031/032` — the 0.6 storage/producer decisions #295 must compose + with or supersede. +- `docs/archive/0.6-storage-design/issue-169-storage-spike.md` — the spike that produced the + #295 direction (§ "The real break point is the claim model"; "A naive claim-ledger + implementation is too slow" → the cursor-allocator conclusion). +- `docs/archive/0.6-storage-design/lease-plane-redesign-spike.md` — companion lease-plane spike. +- `correctness/README.md` — TLA+ model families (what needs re-modeling for #295). +- External: `hardbyte/postgresql-job-queue-benchmarking` — the #169 harness and the cross-system + comparison. + +**Referenced-but-closed context:** #169 (0.6 storage release-blocker investigation), #197 (0.6 +release-readiness tracker), #242 (maintenance-branch split, closed via #302), #290 (live +terminal counter), #314 (SQL-only install/upgrade docs), #359 (queue-storage +`cancel_by_unique_key` fix). diff --git a/docs/0.7-roadmap.md b/docs/0.7-roadmap.md new file mode 100644 index 00000000..0f410f57 --- /dev/null +++ b/docs/0.7-roadmap.md @@ -0,0 +1,439 @@ +# Awa 0.7 — Design & Roadmap + +> **Status:** Proposed. This is the design document for the 0.7 release cycle: strategic +> decisions, workstreams, fully-scoped issues (existing and new), experiments with decision +> rules, milestones, and release gates. Companion context: [`0.7-planning-brief.md`](0.7-planning-brief.md) +> (0.6 recap, open-issue inventory, the #169 hot-row chronology, and the #197 process template). +> No timeline, effort, or cost assumptions are made; sequencing is expressed as dependencies +> and gates, not dates. + +--- + +## 1. Theme and thesis + +**0.6 was the storage release** — it made the substrate honest: bounded degradation under +pinned MVCC horizons, append-only lifecycle, one-way-door migration tooling, and a passed +benchmark gate. + +**0.7 is the operations, observability, and multi-tenancy release** — it makes Awa the +Postgres job queue a serious team can adopt *without a spelunking expedition*: deploy it with +one Helm command, secure it by default, watch a job's trace from enqueue to completion, protect +tenants from each other, chain A→B without a workflow engine, and enqueue from any language. + +The storage-engine rewrite (#295) does **not** headline 0.7. It runs as an **evidence track** +with an explicit decision gate, for three reasons grounded in the 0.6 record: + +1. **The gate passed.** The corrected long-horizon shape holds 798/s through a 60-minute pinned + reader with full dead-tuple reclaim. The burning platform that motivated the RFC in May was + substantially extinguished by rc.1. +2. **The RFC was partly pre-paid.** Its highest-value components — cursor allocation (#321) and + append-only ready segments + tombstones (#323) — already shipped *inside* the current engine + via staged migrations. The 0.6 endgame demonstrated the engine can absorb structural change + in place (v016 → v039) without a new engine identity. +3. **The remaining gap is unquantified.** The visible deltas vs. the rotation-model reference are + WAL/job (~2.2–2.9 KiB vs ~452 B) and the single-shard completion ceiling. Nobody has yet + measured how much of the WAL gap is *intrinsic to Awa's contract* (per-attempt identity, + receipts, terminal evidence, retries) versus *architectural waste*. Rewriting storage twice + in consecutive releases — with restore-only rollback — to chase an unquantified number would + repeat the exact mistake the 0.6 process avoided. + +If the experiments (§5) show a ≥2× win reachable through staged in-place migrations, storage +work enters 0.7 scope through Gate A. Otherwise the RFC graduates to 0.8 with real data behind +it, and 0.7 ships targeted mitigations. + +--- + +## 2. Strategic decisions + +Each decision is stated with its rationale and its reversal condition. These are the calls that +shape everything below. + +### D1 — Storage evolution over storage replacement +**Decision:** The default path for #295's ideas is *evolution of the existing queue-storage +engine via staged migrations (v040+)*, not a third engine identity with its own +`prepare/enter/finalize` transition. A new engine identity is justified only if experiments +prove the lifecycle shape cannot be fixed in place. + +**Rationale:** The transition machinery, the one-way door, and operator retraining are the +expensive parts of an engine swap — 0.5→0.6 proved it. The 0.6 endgame also proved the converse: +v016→v039 restructured lane cursors, ready segments, terminal counts, and claim routing *inside* +the engine, each individually testable, each individually benchmarked. "Fix one hot row, the +next surfaces" is a viable delivery strategy when each fix is a migration; it is not viable when +each fix requires a fleet-wide cutover. + +**Reversal condition:** Gate A (§5, E1/E3) demonstrates that the mutable receipt/lease plane +itself — not any individual table — is the bottleneck, and that no in-place restructuring +closes it. + +### D2 — Canonical engine: gate in 0.7, remove in 0.8 +**Decision:** `awa migrate` for 0.7 **refuses to run unless the cluster's storage state is +`active`** (queue-storage finalized) or the install is fresh. The canonical engine is formally +deprecated in 0.7 (docs, release notes, startup warning) and its claim/execution/trigger paths +are deleted in 0.8. Upgrades therefore step 0.5 → 0.6 (finalize) → 0.7. + +**Rationale:** Canonical exists today only as the transition source. Every feature in this +roadmap would otherwise need dual-engine implementation, dual-engine tests (#360 documented the +cost of *not* doing that), and dual-engine docs. Requiring finalization at the 0.7 boundary +converts #360's dual-engine matrix from a permanent tax into a bounded compatibility suite, and +gives operators one unambiguous instruction. Stepping-stone upgrades are the established norm +(Oban, River, Postgres itself). + +**Reversal condition:** Beta feedback showing a meaningful population stuck mid-transition for +reasons Awa can fix. + +### D3 — Deployment surfaces become products, not patterns +**Decision:** Promote ADR-027 (callback ingress) and ADR-028 (maintenance-only role) from +Proposed to Accepted and ship all four deployment shapes as first-class, documented, +Helm-packaged artifacts: **worker**, **maintenance-only runtime** (#282), **callback ingress** +(`awa callbacks serve`), and **admin UI** — plus the **serverless `tick()`** shape (#118) for +zero-infrastructure deployments. + +**Rationale:** The surface taxonomy in ADR-027 (private admin / public signed callbacks / +internal workers / internal maintenance) is correct and already partially implemented (#291/#293 +shipped the callback-only router). What's missing is the last mile: binaries/entrypoints, health +endpoints, auth, and an installable chart. This is the highest-leverage adoption work in the +tracker. + +### D4 — Multi-tenancy ships tiered, honestly +**Decision:** #340 (per-key execution control) is a headline 0.7 feature, delivered in tiers: +**Tier 1 (committed):** worker-local per-key rate limiting and concurrency caps (approximate +across a fleet, exact per worker, zero storage footprint — documented as such). +**Tier 2 (experiment-gated):** storage-accurate fleet-wide per-key concurrency via claim-time +key-window checks, shipped only if E5 finds a design with **zero new hot mutable rows** and +bounded claim-path cost. + +**Rationale:** The #169 lesson is absolute: no per-key counter family may become a new hot +mutable table. A worker-local limiter captures most real-world value (noisy-neighbor damping, +per-tenant API-budget respect) immediately; the exact fleet-wide semantics are a hard storage +design problem that deserves the same evidence-first treatment as the engine. Shipping Tier 1 +as "approximate, per-worker" with clear docs beats shipping nothing or shipping a lie. + +### D5 — Safe by default on the admin surface +**Decision:** Implement #343 (built-in token auth + documented OIDC proxy pattern), and change +the default posture: **when no auth is configured and the bind address is non-loopback, `awa +serve` starts in read-only mode with a visible banner** explaining how to enable mutation +(configure `AWA_ADMIN_TOKEN` or an auth proxy, or pass `--i-understand-open-admin`). + +**Rationale:** The UI exposes retry, cancel, DLQ purge, batch ops, and storage-transition +controls. "Open mutating admin on 0.0.0.0" should be a choice, never an accident. Loopback +stays frictionless for development. + +### D6 — The public surface gets a written contract +**Decision:** Publish a **stability policy** (`docs/stability.md`) mapping every public surface +to a semver promise: Rust API, Python API, the SQL producer contract (#342 — `insert_job_compat` +frozen as public v1 with cross-language BLAKE3 `unique_key` and shard-hash test vectors), the +HTTP admin API (unblocked by the #143 `awa-api` crate split), metric names/attributes, and the +CLI. Everything not listed is internal and may change in any release. + +**Rationale:** 0.6's `QueueFanout→PartitionedQueue` beta churn was fine *within* a beta series +but showed the absence of a written boundary. Polyglot producers (#342) and UI/API consumers +(#143) cannot exist without one. This document is cheap and permanently valuable. + +### D7 — Job dependencies, not workflows +**Decision:** Ship #14 as minimal A→B chaining built on ADR-029's transactional follow-up +mechanism: `InsertOpts::after(JobRef)` parks B in the deferred backlog in a `waiting_on` state; +A's guarded finalization transactionally promotes B (success) or applies B's declared +`on_parent_failure` policy (`cancel` default | `enqueue_with_context` | `discard`). Exactly one +parent. No fan-in, no fan-out, no DAG type, no workflow state — restated as a non-goal in the +ADR. + +**Rationale:** The promotion mechanism already exists (ADR-029 inserts follow-ups atomically +with the finalize UPDATE); #14 is mostly a parking state and a policy enum. Single-parent A→B +covers the dominant request ("after the PDF, send the email") while the #81 research issue +guards the workflow boundary. + +### D8 — Observability is end-to-end or it isn't done +**Decision:** #110 ships fully in 0.7: W3C `traceparent` captured at enqueue (write-once on the +job row — append-only, no MVCC concern), restored as span parent at claim/dispatch, propagated +across the PyO3 boundary, span status recorded at guarded finalization, linked spans for +retries/callbacks/cron fires. Paired with a new **admin-UI attempt timeline** (claims, receipts, +heartbeats, callbacks, finalizations per attempt) that link-outs to the operator's tracing +backend by trace id, and a **Grafana alert pack** codifying the watch-list from the 0.6 upgrade +guide. + +--- + +## 3. Workstreams and issues + +Priorities: **P0** = release-defining (0.7 does not tag without it), **P1** = strongly +committed, **P2** = opportunistic, **P3** = research/candidate. "NEW-n" issues are drafted by +this roadmap and not yet filed. + +### WS-1 · Foundations (everything else builds on these) + +| Item | Pri | Scope & acceptance | +| --- | --- | --- | +| **#335** CI sharding | P0 | Matrix-sharded Rust tests (own Postgres per shard), cargo-nextest, audit of every >60s test to pin behavior with sub-second configured windows. **Accept:** worst shard <8 min; no real-time production defaults load-bearing in tests. Do first — it multiplies every later item. | +| **#360** Engine-aware test harness | P0 | `AWA_TEST_ENGINE` parameterization (spike branch exists); engine guard for canonical-only tests; fix the 13 triaged test artifacts; CI matrix leg. Under D2 this becomes a *bounded* compat suite: full matrix until 0.8 removes canonical, then retires to the forward-compat matrix. **Accept:** broad suite green under `queue_storage`; `cancel_by_unique_key`-class defects structurally impossible to ship again. | +| **#367** Forward/backward-compat matrix | P0 | Automated: pinned old binaries (0.5.7, 0.6.0) enqueue→claim→complete against a 0.7-migrated schema; 0.7 binary against 0.6 schema (pre-migrate) fails *loudly and legibly*. Closes the "no test runs a genuinely old binary against the newest schema" gap from #360. **Accept:** matrix in nightly; documented support statement. | +| **#143** `awa-api` crate split | P0 | Extract API response types + handlers from `awa-ui`; `JobResponse`-style wrappers everywhere (no `FromRow` structs over HTTP); `awa-cli` depends on `awa-api` without the React build. Prerequisite for D6's HTTP-API stability promise and for #368 health surfaces. | +| **#368** Worker health & readiness endpoints | P0 | Workers currently run **no HTTP server at all**. Add an opt-in, tiny health listener (`AWA_HEALTH_ADDR`): `/healthz` (process live), `/readyz` (DB reachable, schema compatible, claim loop ticking, maintenance heartbeat fresh). Also `awa health --database-url` CLI probe for probe-less environments. Required by #344. **Accept:** k8s liveness/readiness probes work out of the box; documented. | +| **#369** Stability policy (`docs/stability.md`) | P0 | Per D6. Surface-by-surface semver map; deprecation policy (announce in N, warn in N+1, remove in N+2 unless security); what beta/rc mean for each surface. | +| **#370** Canonical deprecation & 0.7 upgrade gate | P0 | Per D2. Migration refuses non-`active` clusters with a message that names the exact 0.6 finalize steps; startup warning on any canonical code path; 0.8 removal plan documented; `docs/upgrade-0.6-to-0.7.md` (short — the storage step is "be finalized"; everything else is additive). | +| Housekeeping | P1 | Verify-and-close #256 (SeaORM shipped), #347 (partitioned queues shipped; fold unresolved open questions into ADR-031 or new issues), triage #346 (nightly failure). File **#383: 0.7 release-readiness tracker** — the #197 analogue, opened on day one with the six gates from §7 as live checklists. | + +### WS-2 · Storage & performance (evidence track — see §5 for experiments) + +| Item | Pri | Scope & acceptance | +| --- | --- | --- | +| **#246** Deadline-rescue claim regression | P0 | First step per the 0.6 record: **reproduce on 0.6.0 final** (E2) — the release decision explicitly said don't chase it unless it reproduces on current main. If it reproduces: name the mechanism (working-set inflation on `lease_claims` is the standing hypothesis), then fix via candidates: deadline-bucketed partial index (needs partitioned-table migration plan), deadline coarsening (round `deadline_at` to reduce index churn), or rescue-scan via segment cursor. Safety rail from the issue stands: per-claim deadline rescue stays default-on. **Accept:** ≤5% throughput overhead with rescue ON at the 1×256 shape, or a named mechanism + documented mitigation if the fix lands in the Gate-A storage work instead. | +| **#295** Storage RFC → Gate A | P0 (decision), scope conditional | Run E1 (allocator bake-off) + E3 (WAL decomposition). Author the draft ADR from results. **Gate A decision rule (§5)** determines whether restructuring migrations enter 0.7 or the RFC graduates to 0.8 with data. Either way #295's RFC questions 1–5 get *answered*, not deferred. | +| **#371** Ring-state metadata striping | P1 | The residual pinned-horizon accumulators after #355 are `lease_ring_state` (~14k dead tuples/hr) and `claim_ring_state` (~3.5k). Apply the ADR-026 discipline (striping or delta-append) to ring bookkeeping. Small, measurable, keeps the "no dominant hot row" invariant tight. **Accept:** long-horizon idle-phase dead tuples for ring-state ≤ noise. | +| **#341** Backpressure | P1 | Per the issue's own design lean: **soft-signal default, opt-in hard rejection.** Concretely: (a) enqueue paths can return a depth signal (`EnqueueOutcome::pressure`) sourced from lane-head cursors (index-only, no scans); (b) `InsertOpts::backpressure: Off | Signal | Reject{limit}` — `Reject` returns a typed error and is documented as *changing transactional-enqueue semantics* (ADR-006 tension made explicit); (c) `PacedProducer` helpers in Rust + Python (the pattern the bench harness already proved externally); (d) metrics `awa.enqueue.backpressure.{signaled,rejected}`. E6 validates defaults. **Accept:** 2×-capacity offered load with paced producer holds bounded depth; hard-reject never enabled implicitly. | +| **#380** Adaptive claimers (experiment-gated) | P2 | E4: bounded controller adjusting per-queue claimers within `[1, max]` on lane-lag signal, opt-in. Addresses the single-shard default-shape ceiling (10k offered / 4k completed) without making users hand-tune `claimers`. Ships only if E4 shows stability under bursty load; otherwise document tuning presets instead. | +| **#303** Maintenance-task split | P2 | Unchanged: telemetry-gated. Check `awa_maintenance_branch_overrun_total` across beta fleets; implement the rescue+promote split only if non-trivial overruns appear; else close wontfix as the issue itself proposes. | + +### WS-3 · Deployment & operations (D3, D5) + +| Item | Pri | Scope & acceptance | +| --- | --- | --- | +| **#282** + ADR-028 Maintenance-only role | P0 | `.maintenance_only()` builder + `awa maintenance run`. Runs promote/rescue/rotate/prune/cleanup/metadata; never claims, never invokes handlers, never dispatches HttpWorker; preserves leader election + shutdown semantics. Promote ADR-028 to Accepted. **Accept:** the issue's acceptance list; chaos nightly includes a maintenance-only + callback-only + HttpWorker topology with zero always-on general workers. | +| **#372** + ADR-027 Callback ingress deployable | P0 | `awa callbacks serve` — the callback receiver as its own process (router from #291/#293), BLAKE3-signed, no admin surface, no UI assets, minimal deps. Promote ADR-027 to Accepted with the surface-taxonomy table as normative docs. **Accept:** public-ingress deployment documented end-to-end (Helm values included); admin UI reachable only privately in the reference topology. | +| **#118** Serverless `tick()` | P0 | `Client::tick(budget) -> TickReport` (every sub-step bounded, e.g. LIMIT 100; returns promoted/rescued/rotated/pruned/dispatched counts) + authenticated `POST /api/tick`. No persistent leader election — short advisory try-locks + SKIP LOCKED, per the issue's lean. Documented drivers: Cloud Scheduler, Vercel cron, GitHub Actions cron, `pg_cron` SQL wrapper. Explicit non-goals restated: no sub-second latency, no LISTEN/NOTIFY, dispatcher doesn't scale to zero *invisibly*. **Accept:** a demo app runs on scheduler-driven ticks alone with correct promotion/rescue; maintenance-gap behavior documented. | +| **#344** Helm chart | P0 | `charts/awa` in-repo, OCI-published, chart-lint + kind smoke in CI. Templates: worker Deployment (optional auth-proxy sidecar), migration Job (helm hook; `awa migrate` as single DDL owner), UI Deployment (readOnly **defaults true** per D5), optional callback-ingress Deployment, optional maintenance-only Deployment, ServiceMonitor. Chart docs state maintenance/cron are leader-elected in-process — **no CronJob**, except the documented tick() variant. Depends on #368 probes. | +| **#343** Admin auth | P0 | Per D5: `AWA_ADMIN_TOKEN` (constant-time compare; header for API, cookie session for UI), documented oauth2-proxy/OIDC pattern with a worked example, non-loopback-unauthenticated ⇒ read-only default + banner, structured audit log line for every mutating action. Authz tiers explicitly deferred. | +| **#373** `awa doctor` | P1 | One command, human + `--json` output: schema/binary version match, storage state + transition blockers, oldest `xact_start` / xmin horizon age, dead-tuple hotspots vs known table families, autovacuum recency, ring rotation health (`skipped_busy`/`blocked` rates), orphaned runtime instances, NOTIFY round-trip check, DLQ depth, failed-retention pruning stats. Subsumes the `awa diagnose mvcc` promised in #169's operational playbook (never shipped). **Accept:** every troubleshooting.md scenario has a corresponding doctor check. | +| **#374** Connection-pooler compatibility | P1 | pgbouncer/pgcat/RDS-Proxy are undocumented today and transaction-pooling silently breaks LISTEN/NOTIFY. Ship: startup NOTIFY self-test → explicit warning + automatic documented polling fallback cadence; docs matrix (session vs transaction mode, pickup-latency deltas from E7); CI leg running the integration suite through pgbouncer. | + +### WS-4 · Observability (D8) + +| Item | Pri | Scope & acceptance | +| --- | --- | --- | +| **#110** End-to-end tracing | P0 | Per D8 and the issue's own plan: `trace_context` captured at enqueue (W3C tracecontext, <200 B, write-once — composes with append-only storage), parent restored at claim, PyO3 propagation (ADR-004 boundary), span status at guarded finalization (run_lease-checked), linked spans for retry/callback/cron, sampler + propagator config. Rust→execution first, Python boundary second, per the issue. Gated by E8: <2% enqueue overhead. | +| **#375** Attempt timeline in UI + trace link-out | P1 | Per-job attempt view assembled from receipts/leases/closures/terminal evidence: claims, heartbeats, snoozes, callback park/resume, finalization — with configured tracing-backend link-out by trace id. Turns the storage engine's evidence trail into operator UX. Depends on #143. | +| **#376** Grafana alert pack | P1 | Codify the 0.6 upgrade-guide watch-list as importable alerts: `xact_start` age, `rotate…skipped_busy` sustained, `prune…blocked` non-zero, queue-lag SLO, DLQ depth delta, `maintenance_branch_overrun_total`, backpressure signals (#341). Ship beside the existing dashboard JSON. | + +### WS-5 · Workload control & flow (D4, D7) + +| Item | Pri | Scope & acceptance | +| --- | --- | --- | +| **#340** Per-key control — Tier 1 | P0 | **ADR-033.** `InsertOpts::concurrency_key` (defaulting to `ordering_key` when unset), per-queue `KeyPolicy { max_in_flight_per_key, rate_per_key }` enforced in the dispatcher: worker-local exact, fleet-approximate (documented formula: fleet cap ≈ per-worker cap × workers). Gated jobs are *not* claimed-then-requeued; the dispatcher holds them un-dispatched within its claimed batch or snoozes with jitter (bounded, measured in E5). Zero new storage. Metrics: `awa.dispatch.key_gated`, per-key fairness gauge. | +| **#340** Per-key control — Tier 2 | P2 (E5-gated) | Fleet-exact concurrency via claim-time key-window check against live receipts/leases keyed by key-hash (read-only check — no counter table). Ships only if E5 shows acceptable claim-path cost and no cursor-monotonicity violation; otherwise documented as future work with E5 data attached. | +| **#14** Job dependencies | P1 | **ADR-034**, per D7: `InsertOpts::after(JobRef)`, `waiting_on` deferred state, transactional promotion on parent finalize via ADR-029 machinery, `on_parent_failure: Cancel | EnqueueWithContext | Discard`. Single parent only; DAG/fan-in/fan-out named non-goals. TLA+ witness for the promote-on-finalize race (parent finalizes concurrently with child cancel). Rust + Python parity, UI shows the dependency edge on both jobs. | +| **#342** SQL producer contract | P1 | Freeze `awa.insert_job_compat` as public v1 (D6): documented signature + semantics versioned against `schema_version`; cross-language test vectors for BLAKE3 `unique_key` derivation and the portable shard hash (`ordering_key`→shard); a conformance script any producer implementation can run against a live schema; `docs/sql-producer-contract.md`. Reference snippets (Node/Go) as *examples*, explicitly unsupported — no official third-language clients (issue's own out-of-scope). | +| **#81 / #83** Research | P3 | Deliver the written recommendations the issues ask for, during 0.7, so 0.8 planning inherits decisions instead of open questions. #81's recommendation should explicitly evaluate whether ADR-034 (A→B) + ADR-029 (follow-ups) already constitute the "very constrained first-class API" endpoint — my expectation is yes, and the workflow-engine boundary holds there. | + +### WS-6 · Developer experience & documentation + +| Item | Pri | Scope & acceptance | +| --- | --- | --- | +| **#377** Docs site | P1 | Versioned docs site (mdBook or equivalent) built from `docs/` in CI, published per release tag. The 25 KB README slims to overview + quickstart + links. Information architecture: Learn (concepts/quickstarts) / Operate (deploy, upgrade, troubleshoot, doctor) / Reference (config, API, SQL contract, metrics, stability policy) / Internals (architecture, ADRs, TLA+). | +| **#378** Python DX pass | P1 | Complete `.pyi` stubs audited against the PyO3 surface with a CI check (mypy/pyright gate on stubs + examples); error-message audit (every raised error names the fix or links docs); `awa-pg` wheel matrix confirmed for current CPython + platform set. | +| **#379** Operations handbook | P1 | Consolidate deployment.md / deploying-on-managed-postgres.md / troubleshooting.md / upgrade guides into one coherent "run Awa in production" narrative on the docs site, including the pooler matrix (#374), MVCC discipline, autovacuum guidance, and a production-readiness checklist. | +| **#381** Example gallery | P2 | Full runnable reference apps (CI-tested like the existing quickstarts): FastAPI + Awa (bridge + callbacks), Django sync bridge, axum + SeaORM, serverless tick() on a cron driver, and a mixed Rust-producer/Python-worker pipeline. | +| **#382** `awa dev` sandbox | P3 | Candidate: one command that runs a disposable Postgres (docker), migrates, starts a demo worker + UI with seeded jobs. Pure onboarding sugar; do only if cheap. | + +--- + +## 4. New ADRs to write + +| ADR | Title | Status target | Source | +| --- | --- | --- | --- | +| 027 | Callback ingress as a deployable surface | Proposed → **Accepted** | #372 | +| 028 | Maintenance-only runtime role | Proposed → **Accepted** | #282 | +| 033 | Per-key execution control (tiered) | Accepted | #340, E5 | +| 034 | Job dependencies (A→B on ADR-029) | Accepted | #14 | +| 035 | Backpressure and producer flow control | Accepted | #341, E6 | +| 036 | Public surface stability policy | Accepted | #369 / D6 | +| 037 | Canonical engine deprecation & removal | Accepted | #370 / D2 | +| 03x | Segment/cursor storage evolution | Draft from E1/E3; status decided at Gate A | #295 | + +--- + +## 5. Experiments + +Each experiment has a hypothesis, method, metrics, and a **decision rule** — the 0.6 lesson is +that gates must be numeric and written down *before* the run. + +**E1 — Claim-allocator bake-off** *(feeds Gate A; answers #295 Q1/Q2)* +Prototypes on the existing harness: **P-a** current engine, tuned (baseline); **P-b** batch-tick +allocator (pgq-style tick ranges + Awa's deferred backlog as the retry sidecar); **P-c** +claim-ledger v2 (per-segment cursor with the indexed short-circuit lesson from #355 — the +spike's naive anti-join is known-wrong). +Shapes: pinned-horizon at 800/s **and 1,600/s**; single-shard 10k/s; skewed-key. +Metrics: complete/s, depth, WAL/job, dead tuples, claim p99, fairness. +*Decision input to Gate A.* + +**E2 — #246 reproduction on 0.6.0** *(feeds the #246 fix)* +Re-run the issue's reproducer (1×256, rescue ON/OFF) on the 0.6.0 tag. If reproduced: attribute +via `pg_stat_io`/waits/`pgstattuple` on `lease_claims`; A/B the candidate fixes. +**Decision rule:** overhead ≤5% ⇒ close with docs; >5% with named mechanism ⇒ fix in 0.7; +mechanism implicates receipt-plane shape ⇒ fold into Gate A scope. + +**E3 — WAL/job decomposition** *(the keystone experiment; answers "is the pgque gap intrinsic?")* +`pg_waldump` attribution of one soak's WAL across record types and table families: lifecycle +inserts vs receipt/claim evidence vs terminal batches vs index maintenance vs ring bookkeeping. +**Decision rule:** if ≥40% of WAL/job is attributable to *removable architecture* (not contract +evidence), storage restructuring has proven headroom → strong Gate A input. If the gap is mostly +contract-intrinsic, the honest answer is "the ~2 KiB/job is the price of per-attempt identity + +receipts" — document it, close that part of #295's motivation, and stop chasing pgque's number. + +**Gate A — storage scope decision** *(consumes E1 + E2 + E3)* +Restructuring migrations enter 0.7 **only if all three hold**: (i) a prototype ≥ parity clean and +strictly better at 1,600/s pinned; (ii) ≥40% WAL/job reduction attributable per E3; (iii) the +change is deliverable as staged in-place migrations (D1) with TLA+ deltas identified. Otherwise: +0.7 ships #246 fix + #371 + tuning presets; the draft ADR + all evidence graduates to 0.8. + +**E4 — Adaptive claimers** *(feeds #380)* +Lag-signal controller vs static settings across steady/bursty/skewed load. +**Decision rule:** no oscillation, no ordering-contract violation, ≥1.5× default-shape completion +⇒ ship opt-in; else ship documented presets only. + +**E5 — Per-key enforcement prototypes** *(feeds #340 Tier 2 / ADR-033)* +(a) worker-local token bucket (Tier 1 baseline), (b) claim-time key-window read-only check, +(c) shard-pinning + per-shard caps. Skewed tenant distributions (Zipf); metrics: fairness +(Jain's index), hot-tenant starvation bound, claim p99 delta, storage write amplification +(**must be zero new hot mutable rows**). +**Decision rule:** (b) or (c) within 10% claim p99 of baseline and zero hot-row footprint ⇒ +Tier 2 ships; else Tier 1 only, data published in the ADR. + +**E6 — Backpressure defaults** *(feeds #341 / ADR-035)* +Paced producer vs naive at 1×/2×/4× capacity; find the depth-target and pacing defaults that +hold bounded depth without throughput sacrifice under 1×. + +**E7 — Pooler matrix** *(feeds #374)* +Integration suite + latency probes through pgbouncer session and transaction modes; quantify +NOTIFY loss and fallback-polling pickup latency; produce the compatibility table. + +**E8 — Tracing overhead** *(gate for #110)* +Enqueue/claim/complete throughput with trace capture on/off, sampled and unsampled. +**Decision rule:** <2% enqueue overhead sampled ⇒ default-on capture with sampling; 2–5% ⇒ +default-off, opt-in; >5% ⇒ redesign storage of the context before shipping. + +--- + +## 6. Milestones + +Gate-sequenced, not date-sequenced. Later milestones may start early where dependencies allow; +a milestone is *done* when its exit criteria hold in CI/nightly, not when its PRs merge. + +**M0 — Foundations.** +#335, #360, #367, #143, #368, #369, #370 groundwork, housekeeping closes, #383 tracker +opened with §7 gates. +*Exit:* worst CI shard <8 min; dual-engine suite green; compat matrix in nightly; `awa-api` +published as a crate; health endpoints merged; stability policy merged. + +**M1 — Evidence & decisions.** +E1–E3 run; Gate A decided and recorded on #295; E5 run; ADR-033/034/035 drafted; #246 mechanism +named. +*Exit:* Gate A recorded with data; every 0.7 ADR in review or accepted; #246 has a fix plan or a +closure with evidence. + +**M2 — Deployment shapes.** +#282, #372 (ADR-027/028 → Accepted), #118, #343, #373, #374, #344 (last — consumes all of +the above). +*Exit:* the four-surface reference topology (private admin, public signed callbacks, workers, +maintenance-only) installable from the Helm chart with kind smoke in CI; tick()-only demo green. + +**M3 — Observability.** +#110 (Rust path → Python boundary), E8 gate, #375, #376. +*Exit:* a job enqueued in a traced FastAPI request shows one connected trace through a Python +worker retry to completion; UI attempt timeline links to it. + +**M4 — Flow & tenancy.** +#340 Tier 1 (+Tier 2 if E5 passed), #14, #341, #342, #81/#83 recommendations written. +*Exit:* ADR-033/034/035 accepted and implemented per their acceptance sections; SQL contract +conformance script green cross-language. + +**M5 — Storage (scope per Gate A).** +Either: targeted track (#246 fix, #371, #380 if E4 passed, tuning presets) — or additionally +the Gate-A restructuring migrations with TLA+ deltas and a fresh long-horizon acceptance run at +the raised bar. +*Exit:* §7 performance gate numbers hold on clean main. + +**M6 — Release train.** +`0.7.0-beta.1` when M0+M2+M3 are done and M4/M5 are merge-complete (judgment-call gates resolve +under beta exposure, per the 0.6 precedent). RC when every §7 gate is green. Stable when the +tracker checklist is fully checked, release notes carry caveated evidence (not universal +claims), and publish mechanics (crates in dependency order, wheels, chart OCI push, Docker, +GitHub release) are green. + +--- + +## 7. Release gates for 0.7.0 stable + +The #383 tracker opens with these as live checklists (the #197 template, six gates): + +1. **Correctness.** TLA+ green including new models: dependency promotion race (ADR-034), + claim-time key gating if Tier 2 ships, allocator/segment deltas if Gate A passed. Trace + witnesses for each new lifecycle path. Model-to-code mapping docs updated. +2. **Performance.** (a) The 0.6 pinned-MVCC long-horizon shape **still passes** — no regression, + ever; (b) #246 shape: ≤5% rescue-ON overhead or named mechanism + shipped mitigation; + (c) partitioned-queue preset sustains ≥9k jobs/s e2e on the reference 24-CPU harness; + (d) tracing overhead within its E8 gate. +3. **Stability.** 14 consecutive green nightlies (chaos + TLA + failover + pooler leg); + dual-engine suite green; compat matrix green. +4. **Operations.** Helm kind-smoke in CI; health endpoints + doctor shipped; upgrade-0.6-to-0.7 + guide with the D2 gate front-and-center; every new surface has troubleshooting entries. +5. **Documentation.** Stability policy published; SQL contract v1 + conformance script; docs + site live and versioned; release notes framed as caveated point-in-time evidence. +6. **Security.** #343 shipped with the D5 default posture; security.md updated for the + four-surface topology; secret scan clean; callback signing docs current. + +--- + +## 8. Risks and mitigations + +| Risk | Mitigation | +| --- | --- | +| Gate A passes and storage work destabilizes the cycle | D1 keeps it staged in-place migrations, each individually benchmarked and revertible pre-release; the raised #169-style bar applies to *every* storage migration, not just the last one; M5 is the only milestone allowed to slip out to 0.8 wholesale. | +| D2 strands un-finalized 0.6 clusters | The gate message names exact finalize steps; `awa doctor` diagnoses transition blockers; beta feedback is the explicit reversal trigger. | +| Tier-1 per-key semantics get mistaken for fleet-exact | Docs state the approximation formula everywhere the knob appears; metrics expose per-worker vs fleet counts; ADR-033 records the tiering rationale. | +| Scope breadth (five P0 workstreams) | M0 is genuinely first (CI speed + harness pay for themselves); everything else is dependency-ordered with per-milestone exit criteria; the beta-first pattern lets judgment calls resolve under exposure instead of blocking the tag. | +| #110 trace storage interacts with future storage changes | trace_context is write-once at enqueue on append-only rows — deliberately shape-agnostic; confirmed in ADR review against the Gate-A design if it proceeds. | +| Nightly flake debt erodes the 14-green gate | #335's real-time-window audit is P0 precisely for this; any new flake gets an issue within 24h per the tracker rules. | + +--- + +## 9. Explicitly out of scope for 0.7 + +- **Workflow engine features**: DAGs, fan-in/fan-out, sagas, workflow state (D7; #81 guards). +- **Official non-Rust/Python client libraries** (#342 ships the contract, not clients). +- **Replica-aware topologies / read-replica routing** (unchanged from the 0.6 decision). +- **Authz granularity tiers** on the admin surface (auth ships; roles deferred, per #343). +- **A third storage-engine identity** absent Gate A (D1). +- **Multi-database / sharded-cluster federation** — one Postgres per Awa remains the model. + +--- + +## 10. Disposition of all 20 open issues + +| Issue | 0.7 disposition | +| --- | --- | +| #295 storage RFC | Evidence track: E1/E3 → Gate A; ADR drafted either way; implementation conditional (WS-2). | +| #360 dual-engine harness | **In**, P0, M0 (bounded by D2). | +| #347 partitioned queues | Verify shipped in 0.6; fold residual open questions into ADR-031; close. | +| #346 nightly failure | Triage in M0 housekeeping; close or spawn a scoped bug. | +| #344 Helm | **In**, P0, M2. | +| #343 UI auth | **In**, P0, M2, with D5 default-posture change. | +| #342 SQL contract | **In**, P1, M4 (under D6). | +| #341 backpressure | **In**, P1, M4 (ADR-035; E6). | +| #340 per-key control | **In**, tiered per D4 (ADR-033; E5), M4. | +| #335 CI sharding | **In**, P0, M0, first. | +| #303 maintenance split | Telemetry-gated during beta; implement or close-wontfix per its own rule. | +| #295-adjacent #246 regression | **In**, P0 (E2 first; fix or fold into Gate A). | +| #282 maintenance-only role | **In**, P0, M2 (ADR-028 → Accepted). | +| #256 SeaORM | Verify shipped (crate exists, publish chain updated); close. | +| #246 deadline-rescue | (see above) | +| #143 awa-api split | **In**, P0, M0. | +| #118 tick() | **In**, P0, M2. | +| #110 tracing | **In**, P0, M3 (full scope — 0.6 shipped only the opt-in propagation subset). | +| #83 ingress adapters | Research deliverable written during 0.7; likely outcome: reference adapters only. | +| #81 orchestration | Research deliverable written during 0.7; expected outcome: ADR-034 + ADR-029 are the boundary. | +| #14 dependencies | **In**, P1, M4 (ADR-034, per D7). | + +New issues filed: **#367–#382** (§3) plus **#383** (release tracker). ADR numbers 033–037 +are claimed by placeholder files in `docs/adr/` per the numbering convention. + +--- + +## 11. What "best possible" means here, in one paragraph + +Awa's differentiation is *earned honesty*: TLA+ models, raised benchmark bars, caveated release +notes, one-way doors documented before they're crossed. 0.7 extends that ethos from the storage +engine to the whole product surface: deployments that are safe by default, claims about +per-key fairness that state their approximation, a storage rewrite that must beat a written +number before it may exist, a public contract document instead of implied stability, and +operator tooling (`doctor`, health, alerts, timelines) that turns the engine's evidence trail +into something a human on call at 3 a.m. can actually use. That — not a features-per-release +count — is what makes an open-source infrastructure project the best in its category. diff --git a/docs/adr/033-per-key-execution-control.md b/docs/adr/033-per-key-execution-control.md new file mode 100644 index 00000000..e28cd68c --- /dev/null +++ b/docs/adr/033-per-key-execution-control.md @@ -0,0 +1,28 @@ +# ADR-033: Per-key execution control (tiered) + +## Status + +Proposed — number claimed; full design tracked in [#340](https://github.com/hardbyte/awa/issues/340) and experiment E5 in [`docs/0.7-roadmap.md`](../0.7-roadmap.md). + +## Context + +Awa controls execution at queue granularity only. Multi-tenant workloads need per-key +concurrency limits ("at most N in-flight per customer") and fairness across keys within a +queue. The binding constraint from #169: no per-key counter family may become a new hot +mutable table under a pinned MVCC horizon. + +## Decision (to be developed) + +Tiered delivery per roadmap decision D4: + +- **Tier 1 (committed):** worker-local per-key rate limiting and concurrency caps via + `InsertOpts::concurrency_key` + per-queue `KeyPolicy` — exact per worker, approximate across + a fleet (documented formula), zero storage footprint. +- **Tier 2 (experiment-gated):** fleet-exact concurrency via claim-time key-window checks, + only if E5 finds a design with zero new hot mutable rows and bounded claim-path cost. + +## Relationship to other ADRs + +Composes with ADR-005 (priority aging), ADR-010 (rate limiting), ADR-011 (weighted +concurrency), ADR-025 (enqueue shards — `concurrency_key` defaults to `ordering_key`), and +ADR-026 (the delta-ledger discipline any per-key state must follow). diff --git a/docs/adr/034-job-dependencies.md b/docs/adr/034-job-dependencies.md new file mode 100644 index 00000000..3702f1c8 --- /dev/null +++ b/docs/adr/034-job-dependencies.md @@ -0,0 +1,27 @@ +# ADR-034: Job dependencies (single-parent A→B on ADR-029) + +## Status + +Proposed — number claimed; full design tracked in [#14](https://github.com/hardbyte/awa/issues/14) per roadmap decision D7 in [`docs/0.7-roadmap.md`](../0.7-roadmap.md). + +## Context + +"Run B after A completes" currently requires the handler to insert the next job manually. +ADR-029 already delivers transactional follow-up enqueues atomically with the parent's guarded +finalization — the mechanism exists; what is missing is a declarative parking state and a +failure policy. + +## Decision (to be developed) + +- `InsertOpts::after(JobRef)` parks B in the deferred backlog in a `waiting_on` state. +- A's guarded finalization transactionally promotes B (success) or applies B's declared + `on_parent_failure` policy: `Cancel` (default) | `EnqueueWithContext` | `Discard`. +- Exactly one parent. Fan-in, fan-out, DAGs, and workflow state are **non-goals** (the PRD's + "not a workflow engine" boundary; #81 guards it). +- TLA+ witness required for the promote-on-finalize race (parent finalizes concurrently with + child cancel). + +## Relationship to other ADRs + +Builds directly on ADR-029 (transactional follow-up jobs); composes with ADR-019 deferred +storage; bounded by the ADR/PRD workflow non-goal. diff --git a/docs/adr/035-backpressure-flow-control.md b/docs/adr/035-backpressure-flow-control.md new file mode 100644 index 00000000..4df9f78e --- /dev/null +++ b/docs/adr/035-backpressure-flow-control.md @@ -0,0 +1,27 @@ +# ADR-035: Backpressure and producer flow control + +## Status + +Proposed — number claimed; full design tracked in [#341](https://github.com/hardbyte/awa/issues/341) and experiment E6 in [`docs/0.7-roadmap.md`](../0.7-roadmap.md). + +## Context + +Nothing stops producers from outrunning completion; the COPY path (ADR-008) makes it trivial +(measured: ~9,982/s enqueue vs ~4,016/s completion, depth 1.26M). Rejecting an enqueue inside a +user transaction (ADR-006) converts a queue-health condition into a business-transaction +failure, so hard rejection can never be the silent default. + +## Decision (to be developed) + +- **Soft-signal default, opt-in hard rejection**: enqueue paths can return a depth signal + (`EnqueueOutcome::pressure`) sourced from lane-head cursors (index-only, no scans); + `InsertOpts::backpressure: Off | Signal | Reject{limit}` with `Reject` returning a typed + error and documented as changing transactional-enqueue semantics. +- `PacedProducer` helpers in Rust and Python (the pattern the benchmark harness already proved + externally). +- Metrics: `awa.enqueue.backpressure.{signaled,rejected}`. + +## Relationship to other ADRs + +Tensions with ADR-006 (transaction surface) made explicit; reads via the #289/#330 cursor +signals; composes with ADR-025/031 throughput levers rather than replacing them. diff --git a/docs/adr/036-public-surface-stability-policy.md b/docs/adr/036-public-surface-stability-policy.md new file mode 100644 index 00000000..fa6bbaae --- /dev/null +++ b/docs/adr/036-public-surface-stability-policy.md @@ -0,0 +1,24 @@ +# ADR-036: Public surface stability policy + +## Status + +Proposed — number claimed; policy text drafted in [`docs/stability.md`](../stability.md) (Draft), +tracked in [#369](https://github.com/hardbyte/awa/issues/369) per roadmap decision D6. + +## Context + +Awa has at least six consumable surfaces — Rust API, Python API, the SQL enqueue contract +(#342), the HTTP admin API (#143), metric names/attributes, and the CLI — with no written +statement of what is stable. Polyglot producers and API consumers cannot exist without one. + +## Decision (to be developed) + +Publish and maintain `docs/stability.md`: a surface-by-surface compatibility map, a +deprecation policy (announce in N, warn in N+1, remove in N+2 barring security), and explicit +internal/unstable designations (storage schema internals, `done_entries` physical layout, +undocumented SQL). + +## Relationship to other ADRs + +Governs the ADR-016 adapter contract and the #342 SQL producer contract; constrains all future +ADRs that touch a listed surface. diff --git a/docs/adr/037-canonical-engine-deprecation.md b/docs/adr/037-canonical-engine-deprecation.md new file mode 100644 index 00000000..6c2cea24 --- /dev/null +++ b/docs/adr/037-canonical-engine-deprecation.md @@ -0,0 +1,25 @@ +# ADR-037: Canonical engine deprecation and removal + +## Status + +Proposed — number claimed; tracked in [#370](https://github.com/hardbyte/awa/issues/370) per roadmap decision D2 in [`docs/0.7-roadmap.md`](../0.7-roadmap.md). + +## Context + +The canonical (row-mutating) engine exists only as the 0.5→0.6 transition source since +queue-storage became the default (ADR-019, 0.6.0). Keeping it alive taxes every feature with +dual-engine implementation, tests (#360), and docs — the tax that shipped the #359 defect. + +## Decision (to be developed) + +- 0.7 `awa migrate` refuses to run unless storage state is `active` (or the install is fresh), + with a refusal message naming the exact 0.6 finalize steps. +- Canonical formally deprecated in 0.7 (release notes, docs, startup warning). +- Claim/execution/trigger paths removed in 0.8; upgrades step 0.5 → 0.6 (finalize) → 0.7. +- Reversal condition: beta feedback showing a meaningful population stuck mid-transition for + reasons Awa can fix. + +## Relationship to other ADRs + +Completes the ADR-019 supersession of the pre-0.6 storage model; bounds the #360 dual-engine +test matrix; interacts with the staged-transition tooling decisions recorded on #180. diff --git a/docs/adr/README.md b/docs/adr/README.md index 12b04af1..bece6b04 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -40,6 +40,11 @@ Template: `Status / Context / Decision / Consequences (positive, negative) / Alt | 030 | [Durable batch operations for operator bulk mutation](030-batch-operations.md) | Accepted | Filter-driven async bulk mutation with preview, progress, cancellation, retention, and maintenance-led execution; v0.6 starts with `set_priority` and `move_queue` | Refines ADR-019/025 operator mutation paths; complements ADR-028 | | 031 | [Partitioned queues](031-partitioned-queues.md) | Accepted | First-class logical queue partitioning over ordinary physical queues, with domain-separated key routing and Python per-job COPY opts | Composes ADR-019/023/026 storage guarantees; refines the ADR-025 sharding interaction | | 032 | [Failed terminal retention floor](032-failed-terminal-retention.md) | Accepted | Queue-storage prune carries in-floor `failed` terminal rows forward into the live segment as wide synthetic rows so they stay retryable for at least `failed_retention`; rows aged past the floor are folded into `queue_terminal_rollups.pruned_failed_count` and surfaced via `QueueCounts.pruned_failed` | Amends ADR-026's one-retention-unit consequence; refines ADR-019 terminal prune path | +| 033 | [Per-key execution control](033-per-key-execution-control.md) | Proposed | Tiered per-key concurrency limits and fairness: worker-local exact / fleet-approximate first, storage-exact only if it needs zero new hot mutable rows (#340) | Composes ADR-005/010/011/025; bound by ADR-026's ledger discipline | +| 034 | [Job dependencies](034-job-dependencies.md) | Proposed | Single-parent A→B chaining: `waiting_on` parking state promoted transactionally by the parent's guarded finalization, with an `on_parent_failure` policy (#14) | Builds on ADR-029; workflow engine remains a non-goal | +| 035 | [Backpressure and flow control](035-backpressure-flow-control.md) | Proposed | Soft depth signals from lane-head cursors by default, opt-in hard rejection, paced-producer helpers (#341) | Makes the ADR-006 transactional-enqueue tension explicit; composes ADR-025/031 | +| 036 | [Public surface stability policy](036-public-surface-stability-policy.md) | Proposed | Surface-by-surface compatibility map and deprecation policy, published as `docs/stability.md` (#369) | Governs ADR-016 and the #342 SQL producer contract | +| 037 | [Canonical engine deprecation](037-canonical-engine-deprecation.md) | Proposed | 0.7 migrate gate requires a finalized queue-storage transition; canonical paths removed in 0.8 (#370) | Completes ADR-019's supersession of the pre-0.6 model | ## Validation artifacts diff --git a/docs/stability.md b/docs/stability.md new file mode 100644 index 00000000..e6629537 --- /dev/null +++ b/docs/stability.md @@ -0,0 +1,56 @@ +# Public Surface Stability Policy + +> **Status: Draft** — introduced by [`0.7-roadmap.md`](0.7-roadmap.md) decision D6, pending +> ADR-036 acceptance. Tracked in [#369](https://github.com/hardbyte/awa/issues/369). Until +> accepted, this document describes intent, not promises. + +Awa is consumed through several distinct surfaces. This document states, per surface, what is +covered by a compatibility promise, what is explicitly internal, and what each release type may +change. Anything not listed here is **internal** and may change in any release without notice. + +## Release types (pre-1.0) + +While Awa is pre-1.0, "minor" releases (0.6 → 0.7) play the semver-major role: + +| Release type | May contain | +| --- | --- | +| **Patch** (0.7.0 → 0.7.1) | Bug fixes only. No breaking changes to any listed surface. No new migrations except to fix a defective migration. | +| **Minor** (0.7 → 0.8) | Breaking changes to listed surfaces are allowed **only** with: a changelog entry under "Breaking changes", a migration path in the upgrade guide, and — where feasible — a deprecation period per the policy below. | +| **Beta / RC** | Surfaces may change between pre-release tags of the same line (the 0.6 `QueueFanout` → `PartitionedQueue` rename is the precedent). Pre-release tags promise nothing to each other; the stable tag promises everything below. | + +## Deprecation policy + +Announce in release N (changelog + doc note), warn at runtime in N+1 where the surface makes a +runtime warning possible, remove in N+2. Security fixes may compress this schedule; the +changelog will say so explicitly. + +## Surfaces + +| Surface | What is covered | What is not | +| --- | --- | --- | +| **Rust API** (`awa`, `awa-model`, `awa-macros`, `awa-testing`, `awa-seaorm`) | All `pub` items documented on docs.rs. Semver-checked in CI (planned: `cargo-semver-checks`). | `#[doc(hidden)]` items; anything behind an unstable feature flag; `awa-worker`/`awa-ui` internals not re-exported by `awa`. | +| **Python API** (`awa-pg`) | Everything exported by `awa/__init__.py` and typed in the `.pyi` stubs (stub/API drift gated in CI per [#378](https://github.com/hardbyte/awa/issues/378)). Async and `_sync` counterparts per ADR-009. | Underscore-prefixed members; the raw PyO3 module layout. | +| **SQL producer contract** | `awa.insert_job_compat(...)` per [#342](https://github.com/hardbyte/awa/issues/342): signature, semantics, BLAKE3 `unique_key` derivation, `ordering_key` → shard hash. Versioned against `awa.schema_version`; conformance script provided. | Every other function, table, view, and trigger in the `awa.*` schema. Direct DML against storage tables is unsupported. | +| **Terminal read surface** | `{schema}.terminal_jobs` (per ADR-026) and the documented public views. | `done_entries` physical layout and all segment/ring/ledger internals — compact batches mean not every completed job is physically a `done_entries` row. | +| **HTTP admin API** | Versioned response types published by the `awa-api` crate ([#143](https://github.com/hardbyte/awa/issues/143)) once split. | Handler internals; the embedded UI's asset paths; any endpoint not documented in `ui-design.md`. | +| **Callback receiver contract** | The signed callback endpoints (ADR-018/021/027): paths under the configured prefix, signature scheme, payload shapes, error mapping. Identical between embedded router and `awa callbacks serve`. | — | +| **CLI** | Documented commands, their exit codes, and `--json` output schemas (e.g. `awa doctor`, `awa storage status`). | Human-readable (non-`--json`) output formatting. | +| **Metrics** | Metric names, types, and attribute keys listed in the telemetry docs. | Attribute *values* cardinality; bucket boundaries (may be tuned in minors with a changelog note). | +| **Storage schema / migrations** | The migration *path*: `awa migrate` upgrades any supported prior version's schema (support window per [#367](https://github.com/hardbyte/awa/issues/367) and ADR-037's gate). | Schema contents. Tables, indexes, functions, and triggers are internal; migrations may reshape them freely. There is no downgrade path. | +| **Configuration** | Documented `QueueConfig`/builder options, Python `start()` kwargs, and `AWA_*` environment variables in `configuration.md`. | Undocumented env vars; internal tuning defaults (may change in minors with a changelog note). | + +## What "supported" means for binary/schema skew + +Rolling deploys create windows where an older binary runs against a newer schema. The support +statement (maintained with [#367](https://github.com/hardbyte/awa/issues/367)): + +- A binary one minor version behind the schema must either work correctly or fail loudly and + legibly — never silently misbehave. +- A newer binary against an older schema refuses to run and names the migration step. +- The 0.7 boundary additionally requires a finalized queue-storage transition (ADR-037). + +## Relationship to release notes + +Every stable release's changelog lists breaking changes against this document's surface list. +If a change breaks something *not* listed here, it is not a breaking change — but if that +surprises users repeatedly, the fix is to amend this document, not to argue. diff --git a/docs/test-plan.md b/docs/test-plan.md index 6d309baa..4360a958 100644 --- a/docs/test-plan.md +++ b/docs/test-plan.md @@ -244,6 +244,64 @@ Concurrent lifecycle benchmark (1 queue × 128 workers, 20K jobs): The formal suite includes passing configs and expected-counterexample configs. The expected-counterexample configs keep historical bugs executable: old dispatch claim, old view trigger, naive segment race, old storage-transition gate, shard-ignorant prune, and deliberate lock-order cycles. Trace configs use a `TraceIncomplete` invariant as a positive witness: a valid trace violates that invariant after TLC consumes every event. +## 0.7 Planned Validation + +Planned test matrix for the 0.7 cycle, mapped to the roadmap ([`0.7-roadmap.md`](0.7-roadmap.md)) and the release gates on the [#383 tracker](https://github.com/hardbyte/awa/issues/383). Rows move into the matrix above as they are implemented. + +### Harness & compatibility + +| # | Test | Rust | Py | Ref | +| --- | --- | --- | --- | --- | +| V1 | Broad integration suite green under `AWA_TEST_ENGINE=queue_storage` | ✓ | ✓ | #360 | +| V2 | Engine guard rejects canonical-only raw-SQL helpers under queue_storage | ✓ | | #360 | +| V3 | Pinned 0.6.0 binary: enqueue→claim→complete→cancel against 0.7 schema | ✓ | ✓ | #367 | +| V4 | 0.7 binary against pre-migrate 0.6 schema fails loudly (message + exit code asserted) | ✓ | | #367 | +| V5 | `awa migrate` refuses non-`active` storage state, names finalize steps | ✓ | | #370 | + +### Deployment & operations + +| # | Test | Rust | Py | Ref | +| --- | --- | --- | --- | --- | +| V6 | `/readyz` flips 503 on DB-down / schema-mismatch / stalled claim loop | ✓ | | #368 | +| V7 | Maintenance-only role: promotes/rescues/rotates, never claims or dispatches | ✓ | ✓ | #282 | +| V8 | Chaos topology with zero general workers (maintenance-only + callback ingress + HttpWorker) | ✓ | | #282, #372 | +| V9 | Callback contract parity: embedded router vs `awa callbacks serve` (shared contract tests) | ✓ | ✓ | #372 | +| V10 | `tick()` bounded steps; scheduler-driven demo promotes/rescues with no resident worker | ✓ | ✓ | #118 | +| V11 | Unauthenticated non-loopback serve → read-only + banner; token auth constant-time; mutation audit line | ✓ | | #343 | +| V12 | `awa doctor` checks cover every troubleshooting.md scenario; `--json` schema asserted | ✓ | | #373 | +| V13 | Integration suite through pgbouncer session mode; transaction mode engages NOTIFY fallback + warning | ✓ | ✓ | #374 | +| V14 | Helm chart kind smoke: four-surface topology installs, probes pass, migration hook runs | ✓ | | #344 | + +### Observability + +| # | Test | Rust | Py | Ref | +| --- | --- | --- | --- | --- | +| V15 | Trace context: enqueue→claim→finalize one connected trace; retries/callbacks/cron linked | ✓ | | #110 | +| V16 | Trace propagation across the PyO3 boundary (Python handler is a child span) | | ✓ | #110 | +| V17 | Tracing overhead A/B within the E8 gate (<2% sampled enqueue cost) | ✓ | | #110 | +| V18 | Attempt-timeline API assembles ordered events across retries incl. callback park/resume | ✓ | | #375 | +| V19 | Alert pack rules lint/import clean | ✓ | | #376 | + +### Flow & tenancy + +| # | Test | Rust | Py | Ref | +| --- | --- | --- | --- | --- | +| V20 | Per-key Tier 1: worker-local cap holds exactly; fleet approximation matches documented formula | ✓ | ✓ | #340 | +| V21 | Per-key fairness under Zipf-skewed tenants (Jain index bound, no starvation) | ✓ | | #340 | +| V22 | A→B promotion is transactional with parent finalize; `on_parent_failure` policies (TLA+ witness + integration) | ✓ | ✓ | #14 | +| V23 | Backpressure: `Signal` surfaces pressure, `Reject` returns typed error, transactional enqueue unaffected by default | ✓ | ✓ | #341 | +| V24 | SQL contract conformance script green; BLAKE3 unique-key + shard-hash cross-language vectors | ✓ | ✓ | #342 | + +### Storage & performance (gate evidence, benchmark harness) + +| # | Test | Ref | +| --- | --- | --- | +| V25 | 0.6 pinned-MVCC long-horizon shape still passes on 0.7 main (798/s through 60-min pin, bounded depth, full reclaim) | #383 Gate 2 | +| V26 | #246 shape: ≤5% rescue-ON overhead at 1×256, or named mechanism + mitigation | #246 / E2 | +| V27 | Ring-state dead tuples reduced to noise on the long-horizon idle phase | #371 | +| V28 | Partitioned-queue preset ≥9k jobs/s e2e on the reference 24-CPU harness | #383 Gate 2 | +| V29 | Gate A evidence: allocator bake-off (E1) + WAL/job decomposition (E3) run ids recorded on #295 | #295 | + ## Running Tests ```bash