Skip to content

Commit de33709

Browse files
Execute the serving cookbook's console transcripts, and mend their rot
The five console transcripts in 06-serving-and-ops.md were the one place the cookbook's guarantee did not reach: recorded once, never re-run, and the page said so out loud. They had already rotted the way unexecuted docs do — /healthz showed version 0.1.0 against an 0.1.1 tree, `grapharc metrics` had grown `events` and `per_phase` fields the page did not show, and `replay | tail -4` was a line short. tests/test_cookbook_serving.py now marks and replays them the way tests/test_cookbook_models.py does its page's. `verified: cli` — the CLI tour, the `--json` pair and the `models <spec>` resolution — is re-run through a real shell as one session and byte-compared, with the page's random run ids mapped to the re-run's and wall-clock spans masked. `verified: cli varies` — `serve`, replayed against a real server on a free port with every curl required to succeed, and `models --check`, which probes the host — runs for success but is not byte-compared, and the /healthz version line it cannot compare is pinned to grapharc.__version__ by name instead. `needs-credentials` — the `agent` command, which always needs a live model — stays unexecuted and must show no output. An unmarked transcript now fails the suite, and the transcripts themselves were re-recorded against the current tree. Fixes #2 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent 0da93a6 commit de33709

2 files changed

Lines changed: 391 additions & 17 deletions

File tree

docs/cookbook/06-serving-and-ops.md

Lines changed: 50 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -11,11 +11,16 @@ re-runs each one and byte-compares it against the output block underneath. No
1111
live model is called anywhere in this file; the one snippet that would need a
1212
key says so and was not run.
1313

14-
The `console` transcripts are the exception, and it is worth naming rather than
15-
leaving to be discovered: they were produced by running the commands shown, but
16-
the test only pairs ```python blocks, so nothing re-checks them on every commit.
17-
Treat a console block as a record of one real run rather than as a guarantee
18-
about the current tree.
14+
The `console` transcripts are held to the same standard, marked the way
15+
`02-models.md` marks its own. `<!-- verified: cli -->` above a transcript means
16+
every command in it is re-run on every commit and byte-compared — run ids are
17+
random and durations are wall-clock, so the test maps the former and masks the
18+
latter, and every other character has to match. `<!-- verified: cli varies -->`
19+
means every command is re-run and must succeed, but the output describes one
20+
machine — `serve` binds a port, `models --check` probes the host — so its bytes
21+
are the record of one real run rather than a guarantee about yours. The one
22+
`agent` invocation always needs a live model; it is marked
23+
`<!-- needs-credentials -->` and nothing here claims to have run it.
1924

2025
Three things to know before you start:
2126

@@ -1241,28 +1246,31 @@ REGISTRY = GraphRegistry({"qa": build_qa})
12411246
```
12421247

12431248
Then, in a shell. This transcript is a real one: the ids and timestamps differ
1244-
per run, and the only editing is the `...` marking where a long JSON body was
1245-
cut — nothing was reworded.
1249+
per run, and the only editing is the `...` marking where a timestamp or a long
1250+
JSON body was cut, plus the session view re-indented for reading — nothing was
1251+
reworded. The test re-runs every command here against a fresh server and
1252+
requires each to succeed; the bytes below are one run's.
12461253

1254+
<!-- verified: cli varies -->
12471255
```console
12481256
$ PYTHONPATH=. grapharc serve --registry mygraphs:REGISTRY --port 8124
12491257
serving grapharc.server on http://127.0.0.1:8124
12501258
graphs : qa
12511259
ctrl-c to stop
12521260

12531261
$ curl -s localhost:8124/healthz
1254-
{"status":"ok","version":"0.1.0","graphs":["qa"]}
1262+
{"status":"ok","version":"0.1.1","graphs":["qa"]}
12551263

12561264
$ curl -s -X POST localhost:8124/sessions -H 'content-type: application/json' \
12571265
-d '{"graph":"qa","input":{"question":"how do budgets work?"}}'
1258-
{"id":"38231fa41b8b4dad","graph":"qa","thread_id":"38231fa41b8b4dad","status":"queued", ...}
1266+
{"id":"bf5ca55bff7b480f","graph":"qa","thread_id":"bf5ca55bff7b480f","status":"queued", ...}
12591267

1260-
$ curl -s localhost:8124/sessions/38231fa41b8b4dad
1268+
$ curl -s localhost:8124/sessions/bf5ca55bff7b480f
12611269
{
1262-
"id": "38231fa41b8b4dad",
1270+
"id": "bf5ca55bff7b480f",
12631271
"graph": "qa",
12641272
"status": "succeeded",
1265-
"run_id": "f3f2554b3959",
1273+
"run_id": "0d9dce7f61c4",
12661274
"result": {
12671275
"question": "how do budgets work?",
12681276
"answer": "Budgets cap iterations, tokens and time."
@@ -1271,9 +1279,9 @@ $ curl -s localhost:8124/sessions/38231fa41b8b4dad
12711279
...
12721280
}
12731281

1274-
$ curl -s localhost:8124/sessions/38231fa41b8b4dad/trace
1275-
{"ts": "...", "run_id": "f3f2554b3959", "graph": "qa", "node": "answer", "phase": "start", "step": 1}
1276-
{"ts": "...", "run_id": "f3f2554b3959", "graph": "qa", "node": "answer", "phase": "end", "step": 1, "state_delta": {"answer": "Budgets cap iterations, tokens and time."}, "duration_ms": 0.91, "tokens": 15}
1282+
$ curl -s localhost:8124/sessions/bf5ca55bff7b480f/trace
1283+
{"ts": "...", "run_id": "0d9dce7f61c4", "thread_id": "bf5ca55bff7b480f", "attempt": 1, "graph": "qa", "node": "answer", "phase": "start", "step": 1}
1284+
{"ts": "...", "run_id": "0d9dce7f61c4", "thread_id": "bf5ca55bff7b480f", "attempt": 1, "graph": "qa", "node": "answer", "phase": "end", "step": 1, "state_delta": {"answer": "Budgets cap iterations, tokens and time."}, "duration_ms": 1.0288769999533542, "tokens": 15}
12771285
```
12781286

12791287
The banner is printed *before* the server blocks, so a script watching stdout
@@ -1579,8 +1587,11 @@ Exit codes are part of the interface: `0` did the job, `1` ran and the answer
15791587
was negative (two runs differed, a run id had no events, no backend was usable),
15801588
`2` could not run at all (missing file, missing component, unknown model spec).
15811589

1582-
A whole session, verbatim:
1590+
A whole session, verbatim (run ids are random per run and durations are
1591+
wall-clock; the test maps the former, masks the latter, and byte-compares every
1592+
other character):
15831593

1594+
<!-- verified: cli -->
15841595
```console
15851596
$ grapharc demo stage1 --trace trace.jsonl
15861597
...
@@ -1612,6 +1623,8 @@ duration_ms: 0.68
16121623
attempts: 1
16131624
termination_reason: target_met
16141625
per_node: {'start': 1, 'plan': 2, 'act': 2, 'verify': 2, 'finish_target_met': 1}
1626+
events: 16
1627+
per_phase: {'start': 8, 'end': 8}
16151628

16161629
$ grapharc viz trace.jsonl 2a47f18064b7
16171630
flowchart TD
@@ -1625,6 +1638,7 @@ flowchart TD
16251638
n6["verify"] --> n7["finish_target_met"]
16261639

16271640
$ grapharc replay trace.jsonl 2a47f18064b7 | tail -4
1641+
pending = []
16281642
8 ok finish_target_met (0.0ms)
16291643
termination_reason = 'target_met'
16301644
8 nodes · 81 tokens · stopped: target_met
@@ -1642,6 +1656,7 @@ Everything downstream (`metrics`, `viz`, `replay`, `diff`) wants that id.
16421656

16431657
`--json` on any of them, and on failures too:
16441658

1659+
<!-- verified: cli -->
16451660
```console
16461661
$ grapharc metrics trace.jsonl 2a47f18064b7 --json
16471662
{
@@ -1655,7 +1670,18 @@ $ grapharc metrics trace.jsonl 2a47f18064b7 --json
16551670
"duration_ms": 0.75,
16561671
"attempts": 1,
16571672
"termination_reason": "target_met",
1658-
"per_node": {"start": 1, "plan": 2, "act": 2, "verify": 2, "finish_target_met": 1}
1673+
"per_node": {
1674+
"start": 1,
1675+
"plan": 2,
1676+
"act": 2,
1677+
"verify": 2,
1678+
"finish_target_met": 1
1679+
},
1680+
"events": 16,
1681+
"per_phase": {
1682+
"start": 8,
1683+
"end": 8
1684+
}
16591685
}
16601686

16611687
$ grapharc metrics nope.jsonl abc --json; echo "exit $?"
@@ -1669,12 +1695,18 @@ exit 2
16691695

16701696
`grapharc models` needs no credentials to answer what a spec *resolves* to:
16711697

1698+
<!-- verified: cli -->
16721699
```console
16731700
$ grapharc models openrouter/anthropic/claude-haiku-4.5
16741701
spec: openrouter/anthropic/claude-haiku-4.5
16751702
backend: openrouter
16761703
model: anthropic/claude-haiku-4.5
1704+
```
1705+
1706+
`--check` reports the machine it is run on:
16771707

1708+
<!-- verified: cli varies -->
1709+
```console
16781710
$ grapharc models --check
16791711
claude-cli usable 'claude' on PATH at /home/you/.local/bin/claude
16801712
credential: claude subscription login (no API key)
@@ -1699,6 +1731,7 @@ count.
16991731
`grapharc agent` is the one command in this list that always needs a model, so
17001732
there is no scripted form of it and nothing here claims to have run it:
17011733

1734+
<!-- needs-credentials -->
17021735
```console
17031736
$ grapharc agent "summarise README.md" --workspace ./work --max-turns 6 --json
17041737
```

0 commit comments

Comments
 (0)