durability: conformance scorecard + journaled LLM calls

README.md

@@ -20,8 +20,11 @@ The current source version is **v0.13.3 (alpha)**. It includes:
   established AST runtime.
 
 The support boundary is deliberately narrow: one POSIX process, one local
-SQLite store, and at-least-once LLM calls across a hard crash. Design records
-and deliberate cuts live in [`docs/design/`](docs/design/).
+SQLite store, and journaled LLM calls that re-execute at most the single
+in-flight call of the interrupted step across a hard crash. Design records
+and deliberate cuts live in [`docs/design/`](docs/design/); the boundary is
+scored against the published seven durable-execution criteria in
+[`docs/durability-conformance.md`](docs/durability-conformance.md).
 
 ```thread
 thread TwoStep {
@@ -196,9 +199,10 @@ finally:
 
 The runtime stays storage-agnostic — `run_durable` hands it a write-through
 trace and a checkpoint callback, so the same executor runs durable or
-ephemeral. Checkpoints are step-level; a crash mid-step re-runs that step,
-finished steps are reused. Replaying a *completed* run returns the stored
-result and makes no model calls. Details:
+ephemeral. Checkpoints are step-level; a crash mid-step re-runs that step (its
+completed model calls replay from the run's journal — only the in-flight call
+re-executes), finished steps are reused. Replaying a *completed* run returns
+the stored result and makes no model calls. Details:
 [`docs/design/phase-2-durability.md`](docs/design/phase-2-durability.md).
 
 ## Control plane (v0.5)
@@ -552,8 +556,9 @@ platform. Each shipped layer keeps the determinism/trace bet:
     parser hardening, authenticated single-node control plane, source/input
     integrity binding, exclusive worker ownership, CAS resume, packaging,
     security gates, and a non-root container. The support boundary is one POSIX
-    process and one local SQLite store; LLM calls remain at-least-once across a
-    hard crash. See [`docs/production.md`](docs/production.md),
+    process and one local SQLite store; journaled LLM calls re-execute at most
+    the single in-flight call across a hard crash. See
+    [`docs/production.md`](docs/production.md),
     [`SECURITY.md`](SECURITY.md), and the
     [semantic comparison](docs/benchmarks/dsl-comparison.md).
 11. **Canonical Workflow IR** *(v0.13, shipped)* — deterministic JSON and

docs/design/HLD.md

@@ -4,8 +4,9 @@
 would make the design document brittle.*
 
 The supported production boundary is one POSIX process and one local SQLite
-store. "Durable" below means step-boundary checkpoints and crash recovery, not
-deterministic event-history replay. See [`../production.md`](../production.md).
+store. "Durable" below means step-boundary checkpoints, per-call model-call
+journaling, and crash recovery, not deterministic event-history replay. See
+[`../production.md`](../production.md).
 
 ## Purpose
 
@@ -84,7 +85,8 @@ JS build (server-rendered HTML with inline CSS, `dashboard.py`).
 | `src/threadlang/llm.py` | Client backends behind a baseline protocol plus optional capabilities: `LLMClient.complete`, `AgentLLMClient.agent_step`, and `RouteLLMClient.route`. `DryRunClient` (deterministic echo + two-phase agent stub), `OpenAICompatClient` (stdlib HTTP), `AnthropicClient` (SDK). |
 | `src/threadlang/tools.py` | The agent execution boundary: `ToolSpec`/`Tool`/`FunctionTool`, `ToolRegistry` allow-list, deterministic built-ins `echo` + `calculator` (AST-walked arithmetic, no `eval`, no `**`, `tools.py`). |
 | `src/threadlang/trace.py` | `TraceEvent(phase, message, data)`, `Trace` alias, `DenialCode` enum. The durable record's unit. |
-| `src/threadlang/store.py` | Durability (L3): `RunStore` (sqlite tables `runs`/`events`/`step_outputs`, WAL/autocommit), canonical definition/input binding with legacy source fencing, bounded queue/retention, CAS resume, write-through traces, step checkpoints, replay, and metrics queries. |
+| `src/threadlang/store.py` | Durability (L3): `RunStore` (sqlite tables `runs`/`events`/`step_outputs`/`llm_journal`, WAL/autocommit), canonical definition/input binding with legacy source fencing, bounded queue/retention, CAS resume, write-through traces, step checkpoints, replay, and metrics queries. |
+| `src/threadlang/journal.py` | Per-call LLM response journal for durable runs: `run_durable` wraps the run's client in `JournaledLLMClient`, which records request fingerprints + responses in `llm_journal` and replays fingerprint-matched responses on resume — a crash re-executes at most the interrupted step's single in-flight model call. |
 | `src/threadlang/control.py` | Control plane workers (L4): exclusive per-store process lock, orphan requeue, atomic claim, source-or-IR execution, per-thread stores, exception-contained worker loops, and readiness state. |
 | `src/threadlang/server.py` | Authenticated stdlib JSON API + dashboard host: source-or-IR `POST /runs`, paginated run queries, metrics, liveness/readiness, Host/origin/body/input admission checks, and HTML views. `serve()` starts the exclusive worker pool and server together. |
 | `src/threadlang/dashboard.py` | Observability (L5): pure `(record, events, metrics) -> HTML` renderers for the run list (with aggregate panel) and per-run trace timeline; everything `html.escape`d; meta-refresh while a run is in flight. |

docs/design/LLD.md

@@ -4,9 +4,10 @@ Refreshed for v0.13.3. The supported boundary is one POSIX process and one local
 SQLite store; see [`../production.md`](../production.md). Historical line
 references elsewhere in this document are explanatory and not API contracts.
 
-> **Refreshed 2026-08-19.** Covers the shipped canonical-IR execution and
-> durable-binding path plus the v0.12 admission, recovery, and ownership
-> hardening. Where this disagrees with the code, the code wins.
+> **Refreshed 2026-08-22.** Covers the shipped canonical-IR execution and
+> durable-binding path, the v0.12 admission, recovery, and ownership
+> hardening, and per-call LLM response journaling on the durable path
+> (`journal.py`). Where this disagrees with the code, the code wins.
 
 ## Module Breakdown
 
@@ -194,15 +195,41 @@ the shared-client `WorkerPool` relies on (`control.py`).
   under `BEGIN IMMEDIATE`. `claim_next_pending()` atomically claims the oldest
   row. `requeue_orphans()` moves restart-stranded sourced/IR runs back to
   `pending`.
-- Events are sequenced and timestamped; step outputs are upserted checkpoints.
-  Per-run and aggregate metrics are folds over those persisted events.
+- Events are sequenced and timestamped; step outputs are upserted checkpoints;
+  journaled model calls are appended with a per-run `call_seq` and looked up
+  by `(run_id, request_fingerprint, occurrence)`. Per-run and aggregate
+  metrics are folds over those persisted events.
 - `run_durable(...)` compiles the current program to canonical IR and binds its
   digest with canonical inputs. The source digest remains metadata and the
   identity fence for legacy rows lacking canonical definition identity. Resume
   verifies stored IR integrity, definition/input identity, IR version, and
   eligible status before it loads checkpoints. A completed run replays without
   model calls; a fresh run moves `created→running`; any execution exception
-  marks `failed`; success marks `completed`.
+  marks `failed`; success marks `completed`. Unless `journal_llm=False`, the
+  run's LLM client is wrapped in `JournaledLLMClient` (`journal.py`) before
+  `run_program` sees it, so a resumed run replays the interrupted step's
+  completed model calls from `llm_journal` and re-executes at most the single
+  in-flight call.
+
+### `journal.py` — per-call LLM response journal
+
+- `JournaledLLMClient(client, store, run_id)` (`journal.py`) — the per-run
+  wrapper `run_durable` installs. It exposes `complete` unconditionally and
+  `route`/`agent_step` only when the wrapped client has them (class-level
+  annotations, conditionally assigned in `__init__`), so the runtime's
+  `getattr` capability probes see exactly the wrapped client's surface.
+- Every call is keyed by `(run_id, request_fingerprint, occurrence)`: the
+  fingerprint is SHA-256 over the canonical JSON of the full request (`kind`
+  + `model` + prompt / options / messages+tools; `ToolCall`-carrying messages
+  and `ToolSpec`s serialize via `dataclasses.asdict`), and `occurrence` is the
+  per-attempt ordinal of that fingerprint, so two identical requests in one
+  run keep distinct rows (`journal.py`).
+- A journal hit replays the recorded response with no provider call —
+  `agent_step` payloads reconstruct an `AgentTurn` (`_agent_turn_from_json`,
+  `journal.py`); a miss calls through and persists request + response JSON. A
+  fresh run_id starts with an empty journal, so first attempts always call
+  live; only resumed runs replay. Tool calls are not journaled and
+  re-execute; exactly-once remains out of scope (`docs/production.md`).
 
 ### `control.py` — worker pool
 
@@ -391,6 +418,11 @@ events (run_id TEXT, seq INTEGER, phase TEXT, message TEXT,
         PRIMARY KEY (run_id, seq))
 step_outputs (run_id TEXT, step_name TEXT, output TEXT,
               PRIMARY KEY (run_id, step_name))
+llm_journal (run_id TEXT, call_seq INTEGER,             -- append order within the run
+             request_fingerprint TEXT,                  -- sha256 of canonical request JSON
+             occurrence INTEGER,                        -- per-attempt ordinal of the fingerprint
+             request_json TEXT, response_json TEXT, created_at TEXT,
+             PRIMARY KEY (run_id, call_seq))
 ```
 
 ### HTTP JSON contracts (`server.py`)
@@ -547,7 +579,7 @@ Programmatic knobs: `AnthropicClient(api_key, max_tokens=1024)`;
 `OpenAICompatClient(base_url, api_key, max_tokens=1024, timeout=120.0)`;
 `WorkerPool(n_workers=2, poll_interval=0.05)`; `serve(host="127.0.0.1",
 port=8765, n_workers=2, llm_client, tools)`; `run_program(tools=...)` /
-`run_durable(run_id=...)`.
+`run_durable(run_id=..., journal_llm=...)`.
 
 Packaging: zero runtime deps (`pyproject.toml`); optional extra
 `anthropic>=0.40,<1.0` (`pyproject.toml`); `requires-python >= 3.11`

docs/durability-conformance.md

@@ -0,0 +1,23 @@
+# ThreadLang durability conformance
+
+A widely-read 2026 critique — [the Diagrid durability critique](https://www.diagrid.io/blog/checkpoints-are-not-durable-execution-why-langgraph-crewai-google-adk-and-others-fall-short-for-production-agent-workflows) — argues that checkpointing is not durable execution, scores LangGraph, CrewAI, and Google ADK as falling short, and defines the term with seven published criteria. ThreadLang's own docs already decline the full label: the README bounds support to one POSIX process, one local SQLite store, and journaled LLM calls that re-execute at most the single in-flight call of the interrupted step across a hard crash, and [`benchmarks/dsl-comparison.md`](benchmarks/dsl-comparison.md) concludes v0.13 is "step-checkpoint durability, not durable replay". This table scores ThreadLang v0.13.3 against the same seven criteria, so the claim and its limits sit in one place. **Met**, **partial**, and **out of scope by design** describe what the code does today; every verdict cites the module or doc section behind it.
+
+| # | Criterion | Verdict | What the code does |
+|---|---|---|---|
+| 1 | Automatic persistence at every step boundary | **met** | `run_durable` wires persistence into execution: a write-through trace lands every `TraceEvent` in the `events` table as it is appended, and the `on_step_complete` hook checkpoints each finished step to `step_outputs`, on an autocommit connection (`src/threadlang/store.py`). No save calls, decorators, or persistence configuration in the program. A checkpoint is written only after output validation and edge resolution succeed, so a bad route label cannot poison it (`src/threadlang/runtime.py`). |
+| 2 | Deterministic replay from an event log | **partial** | The event log is real: persisted, ordered by `seq`, append-only across attempts (`src/threadlang/store.py`; `docs/design/phase-2-durability.md`, decision 3). Replaying a *completed* run returns the stored output with zero model calls (`store.py`, `run_durable`). But resume is checkpoint-resume, not event-history replay: completed steps are reused, and everything after the last checkpoint re-executes — its completed model calls replay from the per-run journal (`src/threadlang/journal.py`), so at most the interrupted step's single in-flight call is made again. This is the "step-checkpoint durability, not durable replay" conclusion of [`benchmarks/dsl-comparison.md`](benchmarks/dsl-comparison.md). |
+| 3 | Implicit activity boundaries | **met** | The step graph the author writes is the activity decomposition — there is no second layer of activity declarations, decorators, or per-step opt-in; every declared step is a persistence and resume boundary (`src/threadlang/runtime.py`, `_run_steps`). The boundary is the whole step: model/tool turns inside an `agent` loop are not individually checkpointed, so a mid-loop crash re-runs the entire step (`docs/design/phase-2-durability.md`, decision 2), and `emit llm` is not a step checkpoint (`docs/production.md`, durability contract). |
+| 4 | Exactly-once side effects | **out of scope by design** | Model calls, including `emit llm`, are journaled per run — across a hard crash at most the interrupted step's single in-flight call repeats — and tool calls in the interrupted step remain at-least-once; exactly-once external effects are out of scope (`docs/production.md`, durability contract). The durable path enforces the mitigation: tools declared `side_effects=True, idempotent=False` are rejected (`src/threadlang/tools.py`, `ToolRegistry.validate_durable`, called from `run_durable`). The store prevents duplicate *execution* of a run — atomic pending claim, CAS resume (`src/threadlang/store.py`) — but suppressing duplicate *effects* is the tool author's declared idempotency contract, not a runtime guarantee. |
+| 5 | Built-in recovery via durable reminders | **partial** | Recovery exists on the queue path: startup acquires the exclusive worker lock and requeues crash-stranded `running` runs to `pending`, and re-dispatching the same id is safe because resume is idempotent (`src/threadlang/control.py`, `WorkerPool`; `src/threadlang/store.py`, `requeue_orphans`; `docs/design/phase-2-durability.md`, decision 4). But nothing fires on its own: there are no durable reminders or timers, a crashed CLI run waits for an operator's `--resume`, and a dead server process recovers only when the server is restarted. |
+| 6 | Linear code with no manual skip logic | **met** | Programs contain no recovery code: the language has no conditionals, and resume is runtime-managed — checkpointed steps are skipped with the skip itself traced, and a resumed `route` step re-derives its edge from the stored label with no model call (`src/threadlang/runtime.py`, `_run_steps`). The author writes a forward-only graph; the only identifier anyone manages is the run id printed by a failed run. |
+| 7 | Distributed rebalancing | **out of scope by design** | The supported boundary is one POSIX process and one local SQLite store (README; `docs/production.md`, supported boundary). Distributed workers, leases across hosts, and network filesystems are explicit non-goals (`docs/production.md`, explicit non-goals). The worker pool is threads in one process; a second process fails startup against `<store>.worker.lock` rather than rebalancing work. |
+
+## What we deliberately do not claim
+
+- **Durable replay.** No Temporal/Dapr-style event-history reconstruction, patch markers, or in-flight version migration (`docs/production.md`, explicit non-goals). Resume reuses step checkpoints and re-executes the remainder; journaled model calls replay, and at most the single in-flight call is made again.
+- **Exactly-once external effects.** Tool calls are at-least-once across a hard crash, a model call repeats at most the single in-flight invocation, and a crash inside a step re-runs that whole step. The runtime enforces declaration (`ToolSpec.side_effects` / `idempotent`) and rejects what it cannot safely replay; it does not deduplicate effects.
+- **Distributed execution.** One process, one store, no cross-host leases or rebalancing. The failure model is process death and restart on one node, not node loss in a fleet.
+- **Self-firing recovery.** No durable reminder or timer fires while the server is down. Recovery is startup orphan-requeue on the queue path and operator `--resume` on the CLI path; detection of a dead process belongs to whatever supervises the process.
+- **Sub-step granularity.** An `agent` step's tool-use loop is one checkpoint unit — per-turn *response journaling* ships (a journaled turn replays rather than re-calling the provider, `src/threadlang/journal.py`), but the checkpoint/resume unit is still the whole step (`docs/design/phase-2-durability.md`, decision 2).
+
+*Scored 2026-08-22 against ThreadLang v0.13.3. Where this doc disagrees with the code, the code wins and this doc is stale.*