From d457da81715136e41aa5fc5aeb1bc962bfb61f7d Mon Sep 17 00:00:00 2001 From: zayJu Date: Thu, 12 Nov 2026 11:44:52 +0800 Subject: [PATCH 1/5] =?UTF-8?q?feat(examples):=20=E5=A4=8D=E6=9D=82?= =?UTF-8?q?=E7=A4=BA=E4=BE=8B=E5=8A=A0=E5=85=A5=E7=9C=9F=E5=AE=9E=E7=9A=84?= =?UTF-8?q?=20Postgres=20=E5=AE=B9=E5=99=A8?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 复杂项目的数据服务通常在容器里。stack-workshop 增加 db 服务：docker compose 前台运行，-p app_${worktree} 让每个 checkout 有自己的项目、网络和卷，port: auto 以 PORT 映射进容器，健康检查在容器内用 pg_isready；api 同时等待 auth/catalog 的 HTTP 与 db 的 TCP，并在 /health 里报告本 worktree SQLite 的迁移数和记录数。 独立的 compose-stack 与 worktree-data 没有独立需求承接：前者成为 db，后者的 ${worktree} 能力已被整栈的 SQLite 与 Compose 项目覆盖，一并删除。 验证：python3 -m unittest scripts.stack_service_test 4 项通过；合并后的清单在隔离 HOME 下 status --json 解析通过。容器路径需要 Docker，本机未运行 daemon，留给 Linux 基准验证。 Co-Authored-By: Claude Code --- examples/compose-stack/README.md | 52 -------- examples/compose-stack/oberth.yaml | 17 --- examples/stack-workshop/.gitignore | 3 + examples/stack-workshop/README.md | 120 +++++++++++++----- .../docker-compose.yml | 0 examples/stack-workshop/oberth.yaml | 30 +++-- examples/stack-workshop/schema.sql | 7 + examples/stack-workshop/service.py | 51 +++++++- examples/worktree-data/.gitignore | 1 - examples/worktree-data/README.md | 84 ------------ examples/worktree-data/api.py | 44 ------- examples/worktree-data/migrate.py | 32 ----- examples/worktree-data/oberth.yaml | 11 -- scripts/stack_service_test.py | 67 ++++++++++ 14 files changed, 231 insertions(+), 288 deletions(-) delete mode 100644 examples/compose-stack/README.md delete mode 100644 examples/compose-stack/oberth.yaml create mode 100644 examples/stack-workshop/.gitignore rename examples/{compose-stack => stack-workshop}/docker-compose.yml (100%) delete mode 100644 examples/worktree-data/.gitignore delete mode 100644 examples/worktree-data/README.md delete mode 100644 examples/worktree-data/api.py delete mode 100644 examples/worktree-data/migrate.py delete mode 100644 examples/worktree-data/oberth.yaml create mode 100644 scripts/stack_service_test.py diff --git a/examples/compose-stack/README.md b/examples/compose-stack/README.md deleted file mode 100644 index 52b1218d..00000000 --- a/examples/compose-stack/README.md +++ /dev/null @@ -1,52 +0,0 @@ -# Compose stack - -Two PostgreSQL containers, one per worktree, with no shared port, project or -database. Run `oberth up --wait` in each checkout and `oberth status` shows the -assigned port and the `pg_isready` health verdict. - -~~~sh -docker pull postgres:16-alpine # first run only -oberth up --wait -oberth status -oberth down -~~~ - -The manifest uses four facts that matter together: - -| Fact | Why | -|---|---| -| `cmd: docker compose ... up` | Foreground Compose gets SIGTERM from `oberth down` and stops its containers. | -| `-p app_${worktree}` | Containers, network and volumes are per checkout. | -| `port: auto` + `"${PORT}:5432"` | option-berth assigns the host port and passes it as `PORT`; Compose interpolates it. | -| `health: command:docker compose ... exec -T db pg_isready` | Health is checked inside the container, not by a host `pg_isready` that may not exist. | - -`oberth down --force` sends SIGKILL. Compose never gets to stop the containers, -so they can survive the run. Use the normal `oberth down` for this stack. - -A normal `down` waits 5 seconds after SIGTERM before escalating. Compose's own -default stop grace is 10 seconds, so a stack that needs longer than 5 seconds to -stop would be killed mid-shutdown and leave its port held. The example sets -`stop_grace_period: 2s`; a heavier stack should keep its stop time inside -oberth's window, or be stopped with `docker compose down` before `oberth down`. - -## One shared stack instead - -A single Compose project shared by every worktree is not a managed service in -each manifest: one worktree's `down` would stop the database another worktree is -using. Declare it as a read-only `machine:` dependency and start it outside -option-berth: - -~~~yaml -machine: - - name: db - port: 5432 -services: - - name: api - cmd: python3 -u api.py - prepare: python3 prepare-db.py # create this worktree's schema/database - env: - DB_NAME: app_${worktree} -~~~ - -The shared stack owns the port; each worktree owns only its database or schema. -`oberth up` and `oberth down` never start or stop a `machine:` entry. diff --git a/examples/compose-stack/oberth.yaml b/examples/compose-stack/oberth.yaml deleted file mode 100644 index dfd69060..00000000 --- a/examples/compose-stack/oberth.yaml +++ /dev/null @@ -1,17 +0,0 @@ -# One Compose project per worktree. `-p app_${worktree}` keeps the containers, -# network and volumes separate; `port: auto` is handed to Compose as PORT, and -# DB_NAME becomes this worktree's database. -# -# `docker compose up` stays in the foreground on purpose: a normal `oberth down` -# sends SIGTERM to the process group, which lets Compose stop the containers. -# `oberth down --force` sends SIGKILL and does not give Compose that chance; use -# it only after stopping the Compose project yourself. -name: compose-stack -services: - - name: db - cmd: docker compose -f docker-compose.yml -p app_${worktree} up - port: auto - health: - command: docker compose -f docker-compose.yml -p app_${worktree} exec -T db pg_isready -U app -d app_${worktree} - env: - DB_NAME: app_${worktree} diff --git a/examples/stack-workshop/.gitignore b/examples/stack-workshop/.gitignore new file mode 100644 index 00000000..fccafb65 --- /dev/null +++ b/examples/stack-workshop/.gitignore @@ -0,0 +1,3 @@ +.berth/ +dist/ +__pycache__/ diff --git a/examples/stack-workshop/README.md b/examples/stack-workshop/README.md index 369a6311..baa0a4f3 100644 --- a/examples/stack-workshop/README.md +++ b/examples/stack-workshop/README.md @@ -1,21 +1,22 @@ -# 复杂一点的整栈 / The Stack Workshop +# 一整栈 / The Stack Workshop -`hello-worktree` 证明一条链路，`worktree-data` 证明每份 checkout 的数据隔离。 -这个示例回答另一个问题：**当项目不再是一个脚本，而是一整栈互相依赖、启动有先有后、 -有的服务根本没有端口时，人和 agent 还能不能只靠一份事实把整栈跑起来、看明白。** +`hello-worktree` 证明一条链路能跑通。这个示例回答另一个问题：**当项目不再是一个脚本， +而是一整栈互相依赖、启动有先有后、有的服务没有端口、数据库还跑在容器里时， +人和 agent 还能不能只靠一份事实把整栈跑起来、看明白。** -它只需要已安装的 `oberth` 和 Python 3；没有 npm、容器、模型账号或 API key。 -所有 HTTP 服务只监听 `127.0.0.1`。 +它需要已安装的 `oberth`、Python 3，以及 **Docker**（`db` 是一个真实的 Postgres 容器）。 +没有 npm、pip、模型账号或 API key；所有 HTTP 服务只监听 `127.0.0.1`。 ## 这份清单里有什么 -十个服务覆盖了大项目最常见的几类麻烦： +十个 Python 服务加一个容器，覆盖大项目最常见的几类麻烦： | 服务 | 形态 | 它让示例证明什么 | |---|---|---| | `auth` | HTTP，启动前预热 4 秒 | 慢启动不是失败；依赖要等它真正就绪 | | `catalog` | HTTP | 普通依赖 | -| `api` | HTTP，`prepare:` 跑数据库迁移，依赖 `auth` + `catalog` | 前置命令、迁移、按服务等待依赖 | +| `db` | Docker Compose 里的 Postgres，`port: auto` | 端口注入容器、容器内健康检查、`down` 让 Compose 自己停容器 | +| `api` | HTTP，`prepare:` 迁移本 worktree 的 SQLite，依赖 `auth` + `catalog` + `db` | 前置命令、迁移、HTTP 与 TCP 两类依赖 | | `gateway` | HTTP，依赖 `api` | 多级依赖，只有上游健康才对外健康 | | `web` | HTTP，`prepare:` 构建 `dist/`，输入是 `src/app.html` 和 `variant.txt` | 生成物不在 Git 里；输入变了要重新构建 | | `search` | HTTP，依赖 `catalog` | 与 `api` 平行的另一条依赖链 | @@ -24,11 +25,13 @@ | `worker` | 无端口，轮询 `api` | 没有端口不等于没有运行 | | `notifier` | 无端口，轮询 `api` | 多个无端口进程各自归属 | -`machine:` 里的 `postgres` 和 `redis` 是**只读引用**：`status` 只说它们在不在听， -`up` / `down` 不会碰它们。示例本身不依赖它们启动，因此没有数据库或容器也能跑通。 +`machine:` 里只剩 `redis`，它是**只读引用**：`status` 只说它在不在听，`up` / `down` +不会碰它。Postgres 不再写在 `machine:` 里，它是上面由本 worktree 拥有并启动的 `db` 服务。 ## 从仓库运行 +先确认 Docker 在跑（`docker compose version` 能返回版本），然后： + ```sh cd examples/stack-workshop oberth up --wait --json @@ -36,34 +39,79 @@ oberth status --json oberth logs worker --once ``` -`auth` 会先打印 `warming up`，几秒后才开始监听；`up --wait` 会等到依赖和健康检查都就绪 -才返回。`worker` 和 `notifier` 没有端口，`status` 仍会列出它们，并在启动后写入 -`.berth//worker.heartbeat` 与 `notifier.heartbeat`。 +`auth` 会先打印 `warming up`，几秒后才开始监听；`db` 会先拉取镜像再启动容器。 +`up --wait` 会等到依赖和健康检查都就绪才返回。`worker` 和 `notifier` 没有端口， +`status` 仍会列出它们，并在启动后写入 `.berth//worker.heartbeat` +与 `notifier.heartbeat`。 -读 `status` 时注意三件事： +读 `status` 时注意四件事： -- 每个带端口的服务都有自己的实际端口；`port: auto` 不需要人分配。 +- 每个带端口的服务都有自己的实际端口，包括 `db`；`port: auto` 不需要人分配。 +- `api` 的 `/health` 只有在 `auth`、`catalog` 健康、`db` 的端口可连时才返回 200； + 同一个响应里还带着本 worktree SQLite 的 `db`、`migrations`、`notes`。 - `worker` / `notifier` 显示为运行中，但没有 `listening on`。 -- `machine:` 两个引用是 `listening` 或 `not listening`，与项目服务的行分开。 +- `machine:` 的 `redis` 是 `listening` 或 `not listening`，与项目服务的行分开。 结束： ```sh -oberth down --force --json +oberth down --json ``` -`down` 只停这份清单所属 worktree 的服务，不删除 `dist/`、`.berth/` 或别处的东西。 +**这个示例要用普通 `down`，不要用 `down --force`。** 普通 `down` 发 SIGTERM， +`docker compose` 收到后自己停止容器；`--force` 直接 SIGKILL，Compose 没有机会收尾， +容器可能留在机器上。普通 `down` 等 5 秒后才升级信号，所以 `docker-compose.yml` 把 +容器的 `stop_grace_period` 设成 2 秒，保证停容器落在 oberth 的窗口内。 + +`down` 只停这份清单所属 worktree 的服务，不删除 `dist/`、`.berth/`、容器卷或别处的东西。 +要回收某个 worktree 的容器数据，由你在该 worktree 内运行 +`docker compose -p app_ down --volumes`；`identity` 来自 `oberth status --json`。 + +## 容器是怎么接进来的 + +`docker-compose.yml` 里有四个相互配合的事实： + +| 事实 | 为什么 | +|---|---| +| `cmd: docker compose ... up` | 前台 Compose 会收到 `oberth down` 的 SIGTERM，并停止自己的容器。 | +| `-p app_${worktree}` | 容器、网络和卷都按 checkout 分开。 | +| `port: auto` + Compose 里的 `"${PORT}:5432"` | option-berth 分配端口并以 `PORT` 传入，Compose 负责映射。 | +| `health: command: ... exec -T db pg_isready` | 健康在容器里检查，不依赖宿主机是否装了 `pg_isready`。 | + +如果数据库是**一套供所有 worktree 共用**的，不要把它声明成受管服务：任一 worktree 的 +`down` 都会停掉别人正在用的数据库。那种写法是用 `machine:` 只读引用，由你在 +option-berth 之外启动它。本示例选择了每 worktree 一套，所以 `db` 是受管服务。 + +## 每个 worktree 一份数据 + +`${worktree}` 展开成当前 checkout 的稳定身份，形如 `wt_0123456789ab`：由 checkout 的 +规范化根路径派生，同一个 worktree 每次启动都相同，两个 worktree 一定不同。示例把它 +同时放进数据目录、库名和 Compose 项目名： + +```yaml +env: + DB_PATH: .berth/${worktree}/app.db # api 的 SQLite + DB_NAME: app_${worktree} # Postgres 容器的库名（Compose 读取） +``` + +`${worktree}` 是标识符，不是路径：把它嵌进相对路径或数据库名。它只在 `prepare`、`cmd` +和 `env` 里展开；其它 `${...}`（例如 `${HOME}`）仍按原样交给命令。它由 checkout 的路径 +派生，所以移动 worktree 目录会得到新身份，旧数据不会被自动搬走。 + +`api` 的 `prepare:` 在每次启动前执行 `migrate.py`，建表并写入一条种子记录；迁移是幂等的， +所以 `restart` 重跑也不会产生第二份数据。`api` 的 `/health` 返回实际数据库路径、迁移数和 +记录数。 ## 只起一个子栈 -大项目里经常只需要一条链路。`up --only` 只启动**点名的服务**；它们声明的依赖会被等待， -但不会自动加入启动集合。所以要起 API 这条链路，要同时点名它依赖的服务： +大项目里经常只需要一条链路。`up --only` 会启动点名的服务及其声明的依赖闭包： ```sh -oberth up --only api,auth,catalog --wait --json +oberth up --only api --wait --json ``` -这会起 `auth`、`catalog` 和 `api`，不会起 `gateway`、`web`、`worker` 或 `notifier`。 +这会起 `auth`、`catalog`、`db` 和 `api`，不会起 `gateway`、`web`、`worker` 或 `notifier`。 +只想起点名的那一个、不展开依赖时加 `--no-deps`，但那样 `api` 会因为没有上游而保持 503。 再用 `oberth status` 确认没被点名的服务仍然停着。 ## 改完代码，跑的是不是新代码 @@ -77,34 +125,36 @@ oberth up --wait --json `restart` 会重跑受影响服务的 `prepare:`；`web` 的 `prepare_inputs` 包含 `src/app.html` 和 `variant.txt`，所以构建会重跑，`dist/version.txt` 变成新值。 -`status` 里 `manifest_runtime_mismatch` 只回答"进程是否符合当前清单"， -不回答"进程是不是用刚改的源码启动的"——后者要靠重启和输出复核。 +`status` 里 `manifest_runtime_mismatch` 只回答“进程是否符合当前清单”， +不回答“进程是不是用刚改的源码启动的”——后者要靠重启和输出复核。 ## 看见两个 worktree 各自独立 在两个由你创建的 Git worktree 里分别进入本示例，各执行一次 `oberth up --wait --json`： -- 两边所有自动端口都不同； +- 两边所有自动端口都不同，包括 `db` 的宿主机端口； +- 两边的 Compose 项目是 `app_wt_…`，容器、网络和卷各一份； - `.berth//app.db` 是两份文件，`worker.heartbeat` 也是两份； -- 只对一边执行 `down`，另一边的服务和无端口 worker 继续运行。 +- 只对一边执行普通 `down`，另一边的服务、容器和无端口 worker 继续运行。 `oberth` 不创建、修改或删除 Git worktree；这些操作仍由你或现有 Git 工具负责。 -## 可重复验收 +## 怎么验证 -从仓库根目录： - -```sh -mage buildEngine -python3 scripts/verify-example.py --binary "$PWD/bin/oberth" -``` +每 PR 的冒烟只跑最小示例 `hello-worktree`，不跑这里，也不碰 Docker。这个复杂示例是 +**手动基准**的输入：基准在 Linux、需要 Docker，比较装了 oberth 与不装 oberth 的 agent +在同一个整栈上的完成情况，不是合并门禁。怎么发起、看什么数，见 +[`docs/agent/agent-eval.md`](../../docs/agent/agent-eval.md)。 -验收脚本在一个一次性 worktree 中复制本示例、启动整栈、检查健康与无端口 worker， -结束后回收自己创建的进程，不接触你日常的 daemon、账本或仓库。 +手工跑通本示例仍然用上面的 `oberth up --wait --json` / `status --json` / 普通 `down --json`； +验收脚本不负责它，避免把 Docker 和 token 成本带进每 PR 的门禁。 ## 常见问题 +- `db` 起不来：先确认 Docker daemon 在跑、`docker compose version` 可用；容器内的 + `pg_isready` 才是健康依据，宿主机不需要装 Postgres 客户端。 - `auth` 还没就绪：这是示例故意的慢启动；`up --wait` 会等，直接 `curl` 太早会得到 503。 - `web` 起不来：先看 `dist/` 是否生成；`prepare:` 的失败会保留在 `oberth logs web --once`。 - 迁移失败：`oberth logs api --once` 保留真实错误；不要用超时当作成功。 +- 端口被残留容器占着：用 `docker compose -p app_ down` 回收，先别杀别的进程。 - 项目在 HOME 外：把示例放在 HOME 下，或明确审阅后使用 `--allow-outside-home`。 diff --git a/examples/compose-stack/docker-compose.yml b/examples/stack-workshop/docker-compose.yml similarity index 100% rename from examples/compose-stack/docker-compose.yml rename to examples/stack-workshop/docker-compose.yml diff --git a/examples/stack-workshop/oberth.yaml b/examples/stack-workshop/oberth.yaml index d596c957..a0b58d7b 100644 --- a/examples/stack-workshop/oberth.yaml +++ b/examples/stack-workshop/oberth.yaml @@ -1,6 +1,7 @@ -# A ten-service stack with a slow dependency, a prepared frontend, a migrated -# database, a portless worker and read-only machine dependencies. Every command -# is explicit: nothing here is inferred from a running process. +# A ten-service Python stack plus one real Postgres container, a slow +# dependency, a prepared frontend, a migrated database, portless workers and a +# read-only machine dependency. Every command is explicit: nothing here is +# inferred from a running process. The stack needs Docker for the `db` service. name: stack-workshop services: - name: auth @@ -20,16 +21,30 @@ services: SERVICE_NAME: catalog VERSION_FILE: variant.txt + # A real Postgres container. `docker compose up` stays in the foreground so a + # normal `oberth down` sends SIGTERM to Compose and Compose stops its own + # containers; `down --force` sends SIGKILL and would leave them running. + # `-p app_${worktree}` gives every checkout its own project, network and + # volume, and `port: auto` is handed to Compose as PORT. + - name: db + cmd: docker compose -f docker-compose.yml -p app_${worktree} up + port: auto + health: + command: docker compose -f docker-compose.yml -p app_${worktree} exec -T db pg_isready -U app -d app_${worktree} + env: + DB_NAME: app_${worktree} + - name: api prepare: python3 migrate.py prepare_inputs: [schema.sql] cmd: python3 -u service.py port: auto health: /health - depends_on: [auth, catalog] + depends_on: [auth, catalog, db] env: SERVICE_NAME: api DB_PATH: .berth/${worktree}/app.db + TCP_REQUIRE: ${db.url} VERSION_FILE: variant.txt REQUIRE: ${auth.url}/health,${catalog.url}/health @@ -97,10 +112,9 @@ services: API_URL: ${api.url} HEARTBEAT: .berth/${worktree}/notifier.heartbeat -# Read-only references. The stack reads them if they exist; `up` and `down` -# never start or stop them. +# Read-only reference. The stack reads it if it exists; `up` and `down` never +# start or stop it. Postgres is no longer a machine reference: it is the `db` +# service above, owned and started per worktree. machine: - - name: postgres - port: 5432 - name: redis port: 6379 diff --git a/examples/stack-workshop/schema.sql b/examples/stack-workshop/schema.sql index 415a3924..bd92d9b7 100644 --- a/examples/stack-workshop/schema.sql +++ b/examples/stack-workshop/schema.sql @@ -1,3 +1,10 @@ +-- Every statement is idempotent: `prepare:` reruns on every up and restart. +CREATE TABLE IF NOT EXISTS schema_migrations ( + id INTEGER PRIMARY KEY, + name TEXT NOT NULL UNIQUE +); +INSERT OR IGNORE INTO schema_migrations (id, name) VALUES (1, '0001_notes'); + CREATE TABLE IF NOT EXISTS notes ( id INTEGER PRIMARY KEY AUTOINCREMENT, body TEXT NOT NULL diff --git a/examples/stack-workshop/service.py b/examples/stack-workshop/service.py index cf8b128d..be773c80 100644 --- a/examples/stack-workshop/service.py +++ b/examples/stack-workshop/service.py @@ -2,18 +2,24 @@ """A small HTTP service used by the stack example. It can warm up before binding (READY_DELAY) and can keep /health at 503 until -the upstream services named in REQUIRE answer. Both are real behaviors a large -stack has: a JVM takes time to listen, and an API is not healthy just because -its own socket is open. +the upstream services named in REQUIRE (HTTP) and TCP_REQUIRE (a raw socket) +answer. Both are real behaviors a large stack has: a JVM takes time to listen, +and an API is not healthy just because its own socket is open. When DB_PATH is +set it also reports the row counts of this worktree's SQLite database, so a +restart that reran `prepare:` can be checked for idempotence. """ from __future__ import annotations import json import os +import socket +import sqlite3 import sys import time from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from pathlib import Path +from urllib.parse import urlsplit from urllib.request import ProxyHandler, build_opener NAME = os.environ.get("SERVICE_NAME", "service") @@ -21,7 +27,9 @@ READY_DELAY = float(os.environ.get("READY_DELAY", "0")) VERSION_FILE = os.environ.get("VERSION_FILE", "variant.txt") REQUIRED_FILE = os.environ.get("REQUIRED_FILE", "") +DB_PATH = os.environ.get("DB_PATH", "") REQUIRE = [item.strip() for item in os.environ.get("REQUIRE", "").split(",") if item.strip()] +TCP_REQUIRE = [item.strip() for item in os.environ.get("TCP_REQUIRE", "").split(",") if item.strip()] OPENER = build_opener(ProxyHandler({})) @@ -36,6 +44,21 @@ def read_version() -> str: VERSION = read_version() +def tcp_target(url: str) -> tuple[str, int]: + """Accept `host:port` or a URL such as `http://127.0.0.1:5432`.""" + parts = urlsplit(url if "://" in url else f"tcp://{url}") + return parts.hostname or "127.0.0.1", parts.port or 0 + + +def wait_for_tcp(url: str) -> bool: + host, port = tcp_target(url) + try: + with socket.create_connection((host, port), timeout=1.0): + return True + except OSError: + return False + + def waiting_on() -> list[str]: waiting = [] for url in REQUIRE: @@ -45,9 +68,28 @@ def waiting_on() -> list[str]: waiting.append(url) except Exception: # noqa: BLE001 - a dependency probe is evidence, not control flow waiting.append(url) + for target in TCP_REQUIRE: + if not wait_for_tcp(target): + waiting.append(target) return waiting +def db_facts() -> dict[str, object]: + if not DB_PATH: + return {} + path = Path(DB_PATH).resolve() + try: + db = sqlite3.connect(path) + try: + migrations = db.execute("SELECT COUNT(*) FROM schema_migrations").fetchone()[0] + notes = db.execute("SELECT COUNT(*) FROM notes").fetchone()[0] + finally: + db.close() + except sqlite3.Error: + return {"db": str(path), "db_error": True} + return {"db": str(path), "migrations": migrations, "notes": notes} + + class Handler(BaseHTTPRequestHandler): def do_GET(self) -> None: # noqa: N802 - BaseHTTPRequestHandler's spelling path = self.path.split("?")[0] @@ -57,7 +99,8 @@ def do_GET(self) -> None: # noqa: N802 - BaseHTTPRequestHandler's spelling body = {"ok": False, "service": NAME, "waiting_on": waiting} elif path in ("/", "/health", "/version"): code = 200 - body = {"ok": True, "service": NAME, "version": VERSION, "pid": os.getpid()} + body = {"ok": True, "service": NAME, "version": VERSION, "pid": os.getpid(), + **db_facts()} else: code = 404 body = {"ok": False, "service": NAME, "path": path} diff --git a/examples/worktree-data/.gitignore b/examples/worktree-data/.gitignore deleted file mode 100644 index 7ecd510b..00000000 --- a/examples/worktree-data/.gitignore +++ /dev/null @@ -1 +0,0 @@ -.berth/ diff --git a/examples/worktree-data/README.md b/examples/worktree-data/README.md deleted file mode 100644 index 86dccc47..00000000 --- a/examples/worktree-data/README.md +++ /dev/null @@ -1,84 +0,0 @@ -# 每个 worktree 一个数据库 / One database per worktree - -这份示例演示 `oberth.yaml` 里的 `${worktree}` 引用：同一个仓库的两个 worktree -同时 `up` 时，各自迁移、各自读写自己的数据库，互不影响；只停一边也不会碰另一边。 - -数据库用 Python 标准库自带的 SQLite，因此和 `hello-worktree` 一样只需要已安装的 -`oberth` 与 Python 3：没有容器、数据库服务、npm/pip 依赖或账号。 - -## 它怎么隔离 - -`${worktree}` 展开成当前 checkout 的稳定身份，形如 `wt_0123456789ab`：由 -checkout 的规范化根路径派生，同一个 worktree 每次启动都相同，两个 worktree -一定不同。示例把身份放进数据目录和库文件名： - -```yaml -env: - DB_PATH: .berth/${worktree}/app.db -``` - -`api` 的 `prepare:` 在每次启动前执行 `migrate.py`，建表并写入一条种子记录； -迁移是幂等的，所以 `restart` 重跑也不会产生第二份数据。`api` 只读自己那份文件， -`/health` 返回实际数据库路径、迁移数和记录数。 - -`${worktree}` 是标识符，不是路径：把它嵌进相对路径（如上）或数据库名 -（如 `app_${worktree}`）。它只在 `prepare`、`cmd` 和 `env` 里展开；其它 -`${...}`（例如 `${HOME}`）仍按原样交给命令。它由 checkout 的路径派生，所以 -移动 worktree 目录会得到新身份，旧数据不会被自动搬走。 - -## 从仓库运行 - -```sh -cd examples/worktree-data -oberth up --wait --json -oberth status --json -curl -s http://localhost:/health -``` - -`/health` 应返回类似： - -```json -{"ok":true,"db":"/path/to/worktree/examples/worktree-data/.berth/wt_…/app.db", - "migrations":1,"notes":1} -``` - -结束服务（不会删除数据）： - -```sh -oberth down --force --json -``` - -## 看见两个 worktree 各自独立 - -在两个由你创建的 Git worktree 里分别进入本示例目录，各执行一次 -`oberth up --wait --json` 和 `status --json`，然后分别读取 `api` 的 `/health`： - -- 两个 `db` 路径不同，且都只包含一次迁移、一条记录； -- 只对其中一个执行 `down`，另一个的 `api` 仍在监听、`/health` 仍可访问； -- 对剩下那个执行 `restart --only api`，迁移不会重复，`migrations` 仍为 `1`。 - -`down` 只停进程，不删除任何一份数据库。要回收某个 worktree 的数据，由你在该 -worktree 内删除 `.berth/`；示例不会自动删除用户数据。 - -`oberth` 本身不创建、修改或删除 Git worktree；这些 Git 写操作仍由你或现有 -Git 工具负责。 - -## 常见问题 - -- `python3` 找不到：先安装 Python 3 并确认该终端能执行 `python3 --version`。 -- 迁移失败：运行 `oberth logs api --once`，真实错误保留在那里，不把超时当作成功。 -- `DB_PATH` 未设置：说明命令不是经 `oberth up` 启动的；直接运行脚本无法知道身份。 -- 项目在 HOME 外：把示例放在你的 HOME 下，或明确审阅后使用 `--allow-outside-home`。 - -## 可重复验收 - -从仓库根目录，先构建，再运行： - -```sh -mage buildEngine -python3 scripts/verify-example.py --binary "$PWD/bin/oberth" -``` - -脚本在两个一次性 Git worktree 中复制并运行本示例（以及 `hello-worktree`）， -验证独立数据库、幂等迁移、单边 `down` 与自动端口，结束后回收自己创建的进程； -它不接触你日常的 daemon、账本或仓库写入。 diff --git a/examples/worktree-data/api.py b/examples/worktree-data/api.py deleted file mode 100644 index 48dc36c7..00000000 --- a/examples/worktree-data/api.py +++ /dev/null @@ -1,44 +0,0 @@ -#!/usr/bin/env python3 -"""Serve a health fact read from this worktree's own SQLite database.""" -import json -import os -import socketserver -import sqlite3 -from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer -from pathlib import Path - -DB_PATH = Path(os.environ["DB_PATH"]).resolve() -PORT = int(os.environ["PORT"]) - - -class Server(ThreadingHTTPServer): - def server_bind(self): - # Avoid HTTPServer's reverse DNS lookup; the address is explicitly local. - socketserver.TCPServer.server_bind(self) - self.server_name = "localhost" - self.server_port = self.server_address[1] - - -class Handler(BaseHTTPRequestHandler): - def do_GET(self): - if self.path != "/health": - self.send_error(404) - return - with sqlite3.connect(DB_PATH) as db: - migrations = db.execute("SELECT COUNT(*) FROM schema_migrations").fetchone()[0] - notes = db.execute("SELECT COUNT(*) FROM notes").fetchone()[0] - body = json.dumps( - {"ok": True, "db": str(DB_PATH), "migrations": migrations, "notes": notes} - ).encode() - self.send_response(200) - self.send_header("Content-Type", "application/json") - self.send_header("Content-Length", str(len(body))) - self.end_headers() - self.wfile.write(body) - - def log_message(self, *_args): - pass - - -if __name__ == "__main__": - Server(("127.0.0.1", PORT), Handler).serve_forever() diff --git a/examples/worktree-data/migrate.py b/examples/worktree-data/migrate.py deleted file mode 100644 index 9fcd43d4..00000000 --- a/examples/worktree-data/migrate.py +++ /dev/null @@ -1,32 +0,0 @@ -#!/usr/bin/env python3 -"""Apply this worktree's schema to its own SQLite database. - -`oberth up` runs this as the api service's `prepare:`, before the long-lived -process starts. DB_PATH comes from the manifest's ${worktree} reference, so -each worktree migrates a different file. Every statement is idempotent, because -a `prepare:` runs again on restart. -""" -import os -import sqlite3 -import sys -from pathlib import Path - - -def main() -> int: - raw = os.environ.get("DB_PATH", "").strip() - if not raw: - print("DB_PATH is not set; run this through `oberth up`", file=sys.stderr) - return 2 - path = Path(raw) - path.parent.mkdir(parents=True, exist_ok=True) - with sqlite3.connect(path) as db: - db.execute("CREATE TABLE IF NOT EXISTS schema_migrations (version INTEGER PRIMARY KEY)") - db.execute("CREATE TABLE IF NOT EXISTS notes (id INTEGER PRIMARY KEY, body TEXT NOT NULL)") - db.execute("INSERT OR IGNORE INTO schema_migrations(version) VALUES (1)") - db.execute("INSERT OR IGNORE INTO notes(id, body) VALUES (1, ?)", (f"seeded in {path}",)) - print(f"migrated {path}", flush=True) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/examples/worktree-data/oberth.yaml b/examples/worktree-data/oberth.yaml deleted file mode 100644 index 4205460d..00000000 --- a/examples/worktree-data/oberth.yaml +++ /dev/null @@ -1,11 +0,0 @@ -# One SQLite database per worktree: ${worktree} is a stable identity for the -# checkout holding this manifest, so two worktrees never share a data file. -name: worktree-data -services: - - name: api - prepare: python3 migrate.py - cmd: python3 -u api.py - port: auto - health: /health - env: - DB_PATH: .berth/${worktree}/app.db diff --git a/scripts/stack_service_test.py b/scripts/stack_service_test.py new file mode 100644 index 00000000..c441ecbc --- /dev/null +++ b/scripts/stack_service_test.py @@ -0,0 +1,67 @@ +"""The stack service's container and database evidence must stay observable.""" +import importlib.util +import os +import socket +import sqlite3 +import tempfile +import unittest +from pathlib import Path +from unittest.mock import patch + +ROOT = Path(__file__).resolve().parent.parent +SOURCE = ROOT / "examples/stack-workshop/service.py" +SCHEMA = ROOT / "examples/stack-workshop/schema.sql" + + +def load_service(**env: str): + defaults = {"SERVICE_NAME": "api", "PORT": "1", "DB_PATH": "", "REQUIRE": "", "TCP_REQUIRE": ""} + with patch.dict(os.environ, {**defaults, **env}, clear=False): + spec = importlib.util.spec_from_file_location("stack_service", SOURCE) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +class StackServiceTests(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.TemporaryDirectory() + self.addCleanup(self.tmp.cleanup) + self.db = Path(self.tmp.name) / "app.db" + connection = sqlite3.connect(self.db) + try: + connection.executescript(SCHEMA.read_text(encoding="utf-8")) + connection.execute("INSERT INTO notes (body) VALUES ('seed')") + connection.commit() + finally: + connection.close() + + def test_tcp_target_accepts_url_and_host_port(self): + service = load_service() + self.assertEqual(service.tcp_target("http://127.0.0.1:5432"), ("127.0.0.1", 5432)) + self.assertEqual(service.tcp_target("127.0.0.1:5432"), ("127.0.0.1", 5432)) + + def test_health_reports_this_worktrees_database(self): + service = load_service(DB_PATH=str(self.db)) + facts = service.db_facts() + self.assertEqual(facts["db"], str(self.db.resolve())) + self.assertEqual(facts["migrations"], 1) + self.assertEqual(facts["notes"], 1) + + def test_unreachable_tcp_dependency_is_not_healthy(self): + with socket.socket() as closed: + closed.bind(("127.0.0.1", 0)) + port = closed.getsockname()[1] + service = load_service(TCP_REQUIRE=f"127.0.0.1:{port}") + self.assertEqual(service.waiting_on(), [f"127.0.0.1:{port}"]) + + def test_reachable_tcp_dependency_is_healthy(self): + with socket.socket() as listener: + listener.bind(("127.0.0.1", 0)) + listener.listen(1) + port = listener.getsockname()[1] + service = load_service(TCP_REQUIRE=f"127.0.0.1:{port}") + self.assertEqual(service.waiting_on(), []) + + +if __name__ == "__main__": + unittest.main() From c38c7e985384f5ae9022223984c3230a880fa312 Mon Sep 17 00:00:00 2001 From: zayJu Date: Thu, 12 Nov 2026 11:45:00 +0800 Subject: [PATCH 2/5] =?UTF-8?q?feat(benchmark):=20=E5=A4=8D=E6=9D=82?= =?UTF-8?q?=E7=A4=BA=E4=BE=8B=E7=9A=84=E6=95=B4=E6=A0=88=E4=BB=BB=E5=8A=A1?= =?UTF-8?q?=E5=8A=A0=E5=85=A5=20Postgres=20=E5=AE=B9=E5=99=A8?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 复杂示例是手动基准的输入，任务要跑的就该是同一份整栈。large 组生成的清单加入 db 服务，判定认识容器健康与端口，native 模式也能从同一份 README 和清单启动它。 基准在 Linux 运行，preflight 现在同时检查 Docker daemon 和 compose 插件，并预拉 postgres:16-alpine；清理阶段按 Compose 标签回收本 run 下的项目，因为 oberth down --force 会 SIGKILL 前台 compose，容器不会自己停。 验证：python3 -m unittest scripts.agent_eval_test scripts.agent_eval_cold_test 132 项通过；--dry-run --suite core 在本机通过；--dry-run --suite large 在本机停在 Docker preflight（daemon 未运行），这是预期的环境要求。 Co-Authored-By: Claude Code --- scripts/agent_eval.py | 28 ++++++++++++++- scripts/agent_eval_lib.py | 36 +++++++++++++++++++ scripts/agent_eval_stacks.py | 70 +++++++++++++++++++++++++++++++----- scripts/agent_eval_test.py | 14 ++++---- 4 files changed, 133 insertions(+), 15 deletions(-) diff --git a/scripts/agent_eval.py b/scripts/agent_eval.py index 2758a3e8..e1a3b2d2 100644 --- a/scripts/agent_eval.py +++ b/scripts/agent_eval.py @@ -75,7 +75,7 @@ write_reference_draft, ) from agent_eval_scenarios import SCENARIOS as CORE_SCENARIOS, Scenario -from agent_eval_stacks import LARGE_SCENARIOS +from agent_eval_stacks import LARGE_SCENARIOS, ensure_image # The core library proves the failure paths on small fixtures. The large library # keeps the same shape but puts the task inside a ten-service stack with slow @@ -185,6 +185,24 @@ def build_identity(binary: Path) -> dict[str, Any]: } +def require_docker() -> None: + """The complex benchmark runs a real Postgres container; no Docker, no run.""" + checks = ( + (["docker", "info", "--format", "{{.ServerVersion}}"], "a running Docker daemon"), + (["docker", "compose", "version"], "the docker compose plugin"), + ) + for argv, what in checks: + try: + done = subprocess.run(argv, capture_output=True, text=True, timeout=30) + except (OSError, subprocess.SubprocessError) as exc: + raise PreflightError(f"the complex benchmark needs {what}: {exc}") from exc + if done.returncode: + raise PreflightError( + f"the complex benchmark needs {what}; `{' '.join(argv)}` failed:\n" + f" {(done.stderr or done.stdout).strip()}" + ) + + def preflight(binary: Path, out_root: Path, pass_env: list[str]) -> None: """Prove the engine can observe listeners before scoring any agent.""" env_root = Path(tempfile.mkdtemp(prefix="oberth-eval-", dir=tempfile.gettempdir())) @@ -1172,6 +1190,14 @@ def main(argv: list[str] | None = None) -> int: print(f"skill: {skill_info['path']} ({skill_info['version']} @ {skill_info['commit']})", file=sys.stderr) if not args.skip_preflight: + if any(SCENARIOS[scenario_id].suite == "large" for scenario_id in scenario_ids): + print("preflight: checking Docker for the complex benchmark...", file=sys.stderr) + try: + require_docker() + ensure_image() + except (PreflightError, RuntimeError) as exc: + print(f"preflight failed:\n{exc}", file=sys.stderr) + return 2 print("preflight: verifying that oberth can observe listeners...", file=sys.stderr) try: preflight(binary, out_root, args.pass_env) diff --git a/scripts/agent_eval_lib.py b/scripts/agent_eval_lib.py index 22ec06c7..e3b894c6 100644 --- a/scripts/agent_eval_lib.py +++ b/scripts/agent_eval_lib.py @@ -822,6 +822,41 @@ def hold_port(self, port: int) -> HeldListener: # -- lifecycle --------------------------------------------------------- + def _cleanup_containers(self) -> list[str]: + """Remove Compose projects this run started, both arms included. + + `oberth down --force` SIGKILLs a foreground `docker compose up`, so the + container survives that stop. Compose labels carry the project and its + working directory; anything under this run's root belongs to the run. + """ + try: + listed = subprocess.run( + ["docker", "ps", "-a", "--format", + '{{.Label "com.docker.compose.project"}}\t{{.Label "com.docker.compose.project.working_dir"}}'], + capture_output=True, text=True, timeout=30, + ) + except (OSError, subprocess.SubprocessError): + return [] + if listed.returncode: + return [f"docker ps failed during cleanup: {listed.stderr.strip()}"] + root = str(self.run_root) + projects: set[tuple[str, str]] = set() + for line in listed.stdout.splitlines(): + project, _, workdir = line.partition("\t") + if project and workdir and (workdir == root or workdir.startswith(root + os.sep)): + projects.add((project, workdir)) + notes = [] + for project, workdir in sorted(projects): + try: + subprocess.run( + ["docker", "compose", "-p", project, "--project-directory", workdir, + "down", "--volumes", "--remove-orphans"], + capture_output=True, text=True, timeout=120, + ) + except (OSError, subprocess.SubprocessError) as exc: + notes.append(f"could not stop compose project {project}: {exc}") + return notes + def cleanup(self, worktrees: dict[str, Path] | None = None) -> list[str]: notes: list[str] = [] worktrees = worktrees or {} @@ -830,6 +865,7 @@ def cleanup(self, worktrees: dict[str, Path] | None = None) -> list[str]: self.oberth(root, ["down", "--force", "--json"], timeout=20) except (OSError, subprocess.SubprocessError): pass + notes.extend(self._cleanup_containers()) known_pids: set[int] = set() for root in worktrees.values(): try: diff --git a/scripts/agent_eval_stacks.py b/scripts/agent_eval_stacks.py index e1dd8635..e28e3c25 100644 --- a/scripts/agent_eval_stacks.py +++ b/scripts/agent_eval_stacks.py @@ -16,6 +16,7 @@ import hashlib import json +import subprocess import time from pathlib import Path from typing import Any, Callable @@ -28,6 +29,7 @@ Verdict, check, free_port, + healthy, port_open, service, up_pid, @@ -37,13 +39,17 @@ EXAMPLE = Path(__file__).resolve().parent.parent / "examples" / "stack-workshop" SOURCE_FILES = ( "service.py", "worker.py", "build.py", "migrate.py", "schema.sql", - "src/app.html", "variant.txt", + "src/app.html", "variant.txt", "docker-compose.yml", ) -PORT_SERVICES = ("auth", "catalog", "api", "gateway", "web", "search", "billing", "ingest") +HTTP_SERVICES = ("auth", "catalog", "api", "gateway", "web", "search", "billing", "ingest") +CONTAINER_SERVICES = ("db",) +PORT_SERVICES = HTTP_SERVICES + CONTAINER_SERVICES WORKER_SERVICES = ("worker", "notifier") SERVICES = PORT_SERVICES + WORKER_SERVICES -API_AND_DEPS = ("auth", "catalog", "api") +API_AND_DEPS = ("auth", "catalog", "db", "api") HEARTBEAT_MAX_AGE_S = 60 +IMAGE = "postgres:16-alpine" +_image_ready = False # --------------------------------------------------------------------------- # @@ -75,6 +81,7 @@ def add( prepare: str | None = None, inputs: tuple[str, ...] = (), env: dict[str, str] | None = None, + health_command: str | None = None, ) -> None: lines.append(f" - name: {service_name}") if prepare: @@ -84,7 +91,11 @@ def add( lines.append(f" cmd: {cmd}") if port: lines.append(f" port: {ports[service_name]}") - lines.append(" health: /health") + if health_command: + lines.append(" health:") + lines.append(f" command: {health_command}") + else: + lines.append(" health: /health") if deps: lines.append(" depends_on: [" + ", ".join(deps) + "]") merged = dict(env or {}) @@ -100,14 +111,22 @@ def add( "VERSION_FILE": "variant.txt", }) add("catalog", env={"SERVICE_NAME": "catalog", "VERSION_FILE": "variant.txt"}) + add( + "db", + cmd=f"docker compose -f docker-compose.yml -p app_{label} up", + health_command=(f"docker compose -f docker-compose.yml -p app_{label} exec -T db " + f"pg_isready -U app -d app_{label}"), + env={"DB_NAME": f"app_{label}"}, + ) add( "api", prepare="python3 migrate.py", inputs=("schema.sql",), - deps=("auth", "catalog"), + deps=("auth", "catalog", "db"), env={ "SERVICE_NAME": "api", "DB_PATH": f".berth/{label}/app.db", + "TCP_REQUIRE": url("db"), "VERSION_FILE": "variant.txt", "REQUIRE": f"{url('auth')}/health,{url('catalog')}/health", }, @@ -158,13 +177,40 @@ def _allocate_ports() -> dict[str, int]: def _machine_ports() -> dict[str, int]: - return {"postgres": free_port(), "redis": free_port()} + return {"redis": free_port()} + + +def ensure_image() -> None: + global _image_ready + if _image_ready: + return + done = subprocess.run(["docker", "pull", IMAGE], capture_output=True, text=True, timeout=300) + if done.returncode: + raise RuntimeError(f"{IMAGE} could not be pulled: {done.stderr.strip()}") + _image_ready = True + + +def _fixture_readme(label: str, ports: dict[str, int]) -> str: + rows = "\n".join(f"| `{name}` | {ports[name]} |" for name in PORT_SERVICES) + return ( + f"# stack-{label}\n\n" + "Every service is declared in `oberth.yaml`, including its env and health check.\n" + "This file gives the raw commands so the same project can be started without\n" + "oberth; the service env comes from the manifest.\n\n" + "| service | port |\n|---|---|\n" + f"{rows}\n\n" + "```sh\n" + f"PORT={ports['db']} DB_NAME=app_{label} " + f"docker compose -f docker-compose.yml -p app_{label} up\n" + "```\n" + ) def _project_files(label: str, ports: dict[str, int], machine: dict[str, int], *, slow_auth: int = 0, extra_env: dict[str, dict[str, str]] | None = None) -> dict[str, str]: files = _source_files() + files["README.md"] = _fixture_readme(label, ports) files[".gitignore"] = ".berth/\ndist/\nconfig/\n" files["scripts/generate_rates.py"] = ( "#!/usr/bin/env python3\n" @@ -302,9 +348,17 @@ def _health_checks(probe: Probe, state: ScenarioState, label: str, services: tup prefix = f"{label}_" if multi else "" checks = [] for name in services: - if name in PORT_SERVICES: + if name in HTTP_SERVICES: key = f"{prefix}{name}_health" checks.append(check(f"{label} {name} answers /health", _ok(probe, key), _http_detail(probe, key))) + elif name in CONTAINER_SERVICES: + doc = probe.statuses.get(label) + item = service(doc, name) + checks.append(check( + f"{label} {name} container is healthy", + healthy(doc, name), + f"status={json.dumps(item)[:200] if item else 'missing'}", + )) else: path = state.effects[f"{prefix}{name}"] ok, detail = _heartbeat(path) @@ -348,7 +402,7 @@ def _urls_for(label: str, ports: dict[str, int], *, multi: bool) -> dict[str, st prefix = f"{label}_" if multi else "" urls = { f"{prefix}{name}_health": f"http://127.0.0.1:{ports[name]}/health" - for name in PORT_SERVICES + for name in HTTP_SERVICES } urls[f"{prefix}api_version"] = f"http://127.0.0.1:{ports['api']}/version" urls[f"{prefix}web_version"] = f"http://127.0.0.1:{ports['web']}/version" diff --git a/scripts/agent_eval_test.py b/scripts/agent_eval_test.py index 138ab153..99124286 100644 --- a/scripts/agent_eval_test.py +++ b/scripts/agent_eval_test.py @@ -749,7 +749,7 @@ def tearDown(self) -> None: def env(self) -> lib.ScenarioEnv: return lib.ScenarioEnv(self.root / "env", self.binary, os.environ.get("PATH", "")) - def test_single_stack_writes_ten_services_and_effects(self) -> None: + def test_single_stack_writes_every_service_and_effect(self) -> None: env = self.env() state = stacks.LARGE_SCENARIOS["stack-start"].setup(env) try: @@ -836,11 +836,12 @@ def test_stack_start_requires_every_service_and_live_heartbeats(self) -> None: self.write_heartbeats(state) scenario = stacks.LARGE_SCENARIOS["stack-start"] http = self.health(version="v1") + doc = status_doc([status_service("db", 7000)]) with mock.patch.object(stacks, "port_open", return_value=True): - good = scenario.judge(probe({"main": status_doc()}, http), state) + good = scenario.judge(probe({"main": doc}, http), state) self.assertTrue(good.completed, [item for item in good.checks if not item["ok"]]) state.effects["worker"].unlink() - bad = scenario.judge(probe({"main": status_doc()}, http), state) + bad = scenario.judge(probe({"main": doc}, http), state) self.assertFalse(bad.completed) def test_stack_subset_uses_single_worktree_effect_keys(self) -> None: @@ -851,8 +852,9 @@ def test_stack_subset_uses_single_worktree_effect_keys(self) -> None: ) scenario = stacks.LARGE_SCENARIOS["stack-subset"] http = self.health(version="v1") + doc = status_doc([status_service("db", 7000)]) with mock.patch.object(stacks, "port_open", return_value=False): - good = scenario.judge(probe({"main": status_doc()}, http), state) + good = scenario.judge(probe({"main": doc}, http), state) self.assertTrue(good.completed, [item for item in good.checks if not item["ok"]]) self.assertTrue(any("worker stayed stopped" in item["name"] for item in good.checks)) # A heartbeat means the worker was started despite the subset task. @@ -861,7 +863,7 @@ def test_stack_subset_uses_single_worktree_effect_keys(self) -> None: json.dumps({"t": time.time(), "service": "worker", "ok": True}) + "\n", encoding="utf-8", ) - bad = scenario.judge(probe({"main": status_doc()}, http), state) + bad = scenario.judge(probe({"main": doc}, http), state) self.assertFalse(bad.completed) def test_stack_stale_requires_new_versions_and_untouched_pids(self) -> None: @@ -955,7 +957,7 @@ def test_same_name_keeps_the_other_checkout_pids(self) -> None: self.write_heartbeats(state, prefix="wt_") http = self.health(prefix="wt_", version="v1") main_doc = status_doc([status_service(name, 4000 + index) for index, name in enumerate(stacks.SERVICES)]) - wt_doc = status_doc() + wt_doc = status_doc([status_service("db", 6000)]) scenario = stacks.LARGE_SCENARIOS["stack-same-name-other-worktree"] with mock.patch.object(stacks, "port_open", return_value=True): good = scenario.judge(probe({"main": main_doc, "wt": wt_doc}, http), state) From d291f228d69500875701ca7528941e4d9b309a1f Mon Sep 17 00:00:00 2001 From: zayJu Date: Thu, 12 Nov 2026 11:45:08 +0800 Subject: [PATCH 3/5] =?UTF-8?q?refactor(ci):=20Examples=20=E9=97=A8?= =?UTF-8?q?=E7=A6=81=E5=8F=AA=E8=B7=91=E6=9C=80=E5=B0=8F=E7=A4=BA=E4=BE=8B?= =?UTF-8?q?=E7=9A=84=E5=86=92=E7=83=9F?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 复杂示例归手动基准后，每 PR 的 Examples 不再跑整栈，也不再需要 Docker。verify-example.py 改名为 smoke.py，只启动 hello-worktree 的两个 worktree，验证自动端口、无端口 worker 和单边停止；workflow 的 job 名保持不变，required checks 不动。 验证：python3 scripts/smoke.py --binary bin/oberth 在 macOS 通过；python3 -m unittest discover -s scripts -p '*_test.py' 270 项通过（1 项跳过）。 Co-Authored-By: Claude Code --- .github/workflows/examples.yml | 8 +- examples/hello-worktree/README.md | 2 +- scripts/smoke.py | 76 ++++++ scripts/verify-example.py | 384 ------------------------------ 4 files changed, 82 insertions(+), 388 deletions(-) create mode 100644 scripts/smoke.py delete mode 100644 scripts/verify-example.py diff --git a/.github/workflows/examples.yml b/.github/workflows/examples.yml index ff836ce5..411a8097 100644 --- a/.github/workflows/examples.yml +++ b/.github/workflows/examples.yml @@ -8,7 +8,7 @@ on: paths: - 'examples/**' - 'engine/**' - - 'scripts/verify-example.py' + - 'scripts/smoke.py' - '.github/workflows/examples.yml' workflow_dispatch: permissions: @@ -33,11 +33,13 @@ jobs: with: go-version-file: engine/go.mod cache-dependency-path: engine/go.sum - - name: Verify the checked-in examples in two worktrees + # The minimal sample only: no Docker, no benchmark. The complex sample + # runs in the manual benchmark, which is the only consumer that needs it. + - name: Run the per-PR smoke check shell: bash run: | set -euo pipefail output=$(mktemp -d) trap 'rm -rf "$output"' EXIT (cd engine && go build -o "$output/oberth" ./cmd/oberth) - python3 scripts/verify-example.py --binary "$output/oberth" + python3 scripts/smoke.py --binary "$output/oberth" diff --git a/examples/hello-worktree/README.md b/examples/hello-worktree/README.md index 9ff761b0..8c4119f6 100644 --- a/examples/hello-worktree/README.md +++ b/examples/hello-worktree/README.md @@ -45,7 +45,7 @@ oberth down --force --json ```sh mage buildEngine -python3 scripts/verify-example.py --binary "$PWD/bin/oberth" +python3 scripts/smoke.py --binary "$PWD/bin/oberth" ``` 脚本复制的就是本目录文件，不再维护另一份类似示例。它验证双 worktree、健康、自动端口、 diff --git a/scripts/smoke.py b/scripts/smoke.py new file mode 100644 index 00000000..347bd417 --- /dev/null +++ b/scripts/smoke.py @@ -0,0 +1,76 @@ +#!/usr/bin/env python3 +"""The per-PR smoke check: run the minimal sample in two disposable worktrees. + +Never touches the caller's daemon, HOME or BERTH_HOME. The complex sample +(`examples/stack-workshop`) is not run here; it is the input of the manual +benchmark, which is the only consumer that needs Docker. +""" +from __future__ import annotations + +import argparse +import importlib.util +import json +import shlex +import shutil +import sys +import tempfile +from pathlib import Path +from urllib.request import build_opener, ProxyHandler + + +def trial_module(): + source = Path(__file__).with_name("verify-e2e.py") + spec = importlib.util.spec_from_file_location("berth_trial", source) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def smoke(module, binary: Path, sample: Path) -> dict: + with tempfile.TemporaryDirectory(prefix="bsmoke-", dir="/tmp") as tmp: + trial = module.Trial(binary, Path(tmp).resolve()) + try: + trial.provision() + for root in trial.roots: + for name in ("api.py", "worker.py", "oberth.yaml"): + shutil.copyfile(sample / name, root / name) + manifest = root / "oberth.yaml" + data = manifest.read_text() + for name in ("api", "worker"): + command = shlex.quote(sys.executable) + f" -u {name}.py" + data = data.replace(f"cmd: python3 -u {name}.py", "cmd: " + json.dumps(command)) + manifest.write_text(data) + trial.command(root, "up", "--wait", "--json") + http = build_opener(ProxyHandler({})) + ports = [] + for root in trial.roots: + state = trial.ready(root) + port = next(s["port_actual"] for s in state["worktree"]["services"] if s["name"] == "api") + ports.append(port) + with http.open(f"http://127.0.0.1:{port}/health", timeout=3) as response: + assert json.load(response) == {"service": "api", "ok": True, "port": port} + assert ports[0] != ports[1], "worktrees received the same port" + trial.command(trial.roots[0], "down", "--force", "--json") + state = trial.status(trial.roots[0]) + assert not any(s["running"] for s in state["worktree"]["services"]), state + trial.ready(trial.roots[1]) + with http.open(f"http://127.0.0.1:{ports[1]}/health", timeout=3) as response: + assert response.status == 200 + return {"sample": "hello-worktree", "worktrees": 2, + "distinct_ports": True, "healthy": True, "portless_worker": True, + "single_side_stop": "passed"} + finally: + trial.close() + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--binary", type=Path, required=True) + binary = parser.parse_args().binary.resolve(strict=True) + sample = Path(__file__).resolve().parent.parent / "examples" / "hello-worktree" + print(json.dumps(smoke(trial_module(), binary, sample))) + print("isolated smoke processes and temporary files cleaned up") + + +if __name__ == "__main__": + main() diff --git a/scripts/verify-example.py b/scripts/verify-example.py deleted file mode 100644 index f7116412..00000000 --- a/scripts/verify-example.py +++ /dev/null @@ -1,384 +0,0 @@ -#!/usr/bin/env python3 -"""Run the checked-in examples in two disposable worktrees, never the user's daemon.""" -from __future__ import annotations - -import argparse -import importlib.util -import json -import shlex -import shutil -import subprocess -import sys -import tempfile -import time -from pathlib import Path -from urllib.request import build_opener, ProxyHandler - - -def trial_module(): - source = Path(__file__).with_name("verify-e2e.py") - spec = importlib.util.spec_from_file_location("berth_trial", source) - module = importlib.util.module_from_spec(spec) - spec.loader.exec_module(module) - return module - - -def verify_hello(module, binary: Path, example: Path) -> dict: - with tempfile.TemporaryDirectory(prefix="bhello-", dir="/tmp") as tmp: - trial = module.Trial(binary, Path(tmp).resolve()) - try: - trial.provision() - for root in trial.roots: - for name in ("api.py", "worker.py", "oberth.yaml"): - shutil.copyfile(example / name, root / name) - manifest = root / "oberth.yaml" - data = manifest.read_text() - for name in ("api", "worker"): - command = shlex.quote(sys.executable) + f" -u {name}.py" - data = data.replace(f"cmd: python3 -u {name}.py", "cmd: " + json.dumps(command)) - manifest.write_text(data) - trial.command(root, "up", "--wait", "--json") - http = build_opener(ProxyHandler({})) - ports = [] - for root in trial.roots: - state = trial.ready(root) - port = next(s["port_actual"] for s in state["worktree"]["services"] if s["name"] == "api") - ports.append(port) - with http.open(f"http://127.0.0.1:{port}/health", timeout=3) as response: - assert json.load(response) == {"service": "api", "ok": True, "port": port} - assert ports[0] != ports[1], "worktrees received the same port" - trial.command(trial.roots[0], "down", "--force", "--json") - state = trial.status(trial.roots[0]) - assert not any(s["running"] for s in state["worktree"]["services"]), state - trial.ready(trial.roots[1]) - with http.open(f"http://127.0.0.1:{ports[1]}/health", timeout=3) as response: - assert response.status == 200 - return {"example": "hello-worktree", "worktrees": 2, - "distinct_ports": True, "healthy": True, "portless_worker": True, - "single_side_stop": "passed"} - finally: - trial.close() - - -def verify_worktree_data(module, binary: Path, example: Path) -> dict: - """Two worktrees migrate and serve separate databases, and one down leaves - the other's process and data alone.""" - http = build_opener(ProxyHandler({})) - - def health(root: Path, timeout: float = 15) -> tuple[int, dict]: - deadline = time.monotonic() + timeout - last = None - while time.monotonic() < deadline: - services = trial.status(root)["worktree"]["services"] - api = next(s for s in services if s["name"] == "api") - if api.get("running") and api.get("port_actual"): - try: - with http.open(f"http://127.0.0.1:{api['port_actual']}/health", timeout=3) as response: - return api["port_actual"], json.load(response) - except OSError as error: - last = error - else: - last = api - time.sleep(.15) - raise RuntimeError(f"api did not become healthy in {root}: {last}") - - with tempfile.TemporaryDirectory(prefix="bwdata-", dir="/tmp") as tmp: - trial = module.Trial(binary, Path(tmp).resolve()) - try: - trial.provision() - for root in trial.roots: - for name in ("api.py", "migrate.py", "oberth.yaml"): - shutil.copyfile(example / name, root / name) - manifest = root / "oberth.yaml" - data = manifest.read_text() - for line, script in (("prepare: python3 migrate.py", "migrate.py"), - ("cmd: python3 -u api.py", "api.py")): - command = shlex.quote(sys.executable) + f" {script}" - data = data.replace(line, line.split(":")[0] + ": " + json.dumps(command)) - manifest.write_text(data) - trial.command(root, "up", "--wait", "--json") - - first = [health(root) for root in trial.roots] - ports, docs = [port for port, _ in first], [doc for _, doc in first] - assert ports[0] != ports[1], f"worktrees share an auto port: {ports}" - assert docs[0]["db"] != docs[1]["db"], f"worktrees share a database: {docs}" - for doc in docs: - assert doc["ok"] is True, doc - assert doc["migrations"] == 1 and doc["notes"] == 1, doc - - # A restart reruns `prepare:`. The migration is idempotent, and the - # other worktree's database is not touched. - trial.command(trial.roots[0], "restart", "--only", "api", "--json") - port0, doc0 = health(trial.roots[0]) - assert port0 == ports[0], f"restart moved the auto port: {ports[0]} -> {port0}" - assert doc0["db"] == docs[0]["db"] and doc0["migrations"] == 1 and doc0["notes"] == 1, doc0 - assert health(trial.roots[1])[1] == docs[1] - - # `down` is scoped to the manifest's own worktree: the other side - # keeps running, and neither data file is deleted. - trial.command(trial.roots[0], "down", "--force", "--json") - state = trial.status(trial.roots[0]) - assert not any(s["running"] for s in state["worktree"]["services"]), state - assert health(trial.roots[1])[1] == docs[1] - assert Path(docs[0]["db"]).is_file() and Path(docs[1]["db"]).is_file() - - return {"example": "worktree-data", "worktrees": 2, - "distinct_ports": True, "distinct_databases": True, - "migrations_per_worktree": 1, "idempotent_restart": True, - "single_side_stop": "passed"} - finally: - trial.close() - - -def verify_stack(module, binary: Path, example: Path) -> dict: - """One full ten-service stack: ordered dependencies, a slow auth, a prepared - frontend, a migration and two portless workers. Runs in one disposable - worktree so the required Examples job stays bounded.""" - http = build_opener(ProxyHandler({})) - - def get_json(port: int, path: str, timeout: float = 20) -> dict: - deadline = time.monotonic() + timeout - last = None - while time.monotonic() < deadline: - try: - with http.open(f"http://127.0.0.1:{port}{path}", timeout=3) as response: - return json.load(response) - except OSError as error: - last = error - time.sleep(.15) - raise RuntimeError(f"{path} on {port} did not answer: {last}") - - with tempfile.TemporaryDirectory(prefix="bstack-", dir="/tmp") as tmp: - trial = module.Trial(binary, Path(tmp).resolve()) - try: - trial.provision() - root = trial.roots[0] - for source in example.rglob("*"): - if not source.is_file() or source.name == "README.md": - continue - target = root / source.relative_to(example) - target.parent.mkdir(parents=True, exist_ok=True) - shutil.copyfile(source, target) - manifest = root / "oberth.yaml" - data = manifest.read_text() - py = sys.executable - replacements = { - "cmd: python3 -u service.py": "cmd: " + json.dumps(f"{py} -u service.py"), - "cmd: python3 -u worker.py": "cmd: " + json.dumps(f"{py} -u worker.py"), - "prepare: python3 migrate.py": "prepare: " + json.dumps(f"{py} migrate.py"), - "prepare: python3 build.py": "prepare: " + json.dumps(f"{py} build.py"), - } - for old, new in replacements.items(): - data = data.replace(old, new) - manifest.write_text(data) - trial.command(root, "up", "--wait", "--wait-timeout", "25s", "--json") - - state = trial.status(root) - services = {item["name"]: item for item in state["worktree"]["services"]} - expected = { - "auth", "catalog", "api", "gateway", "web", "search", "billing", - "ingest", "worker", "notifier", - } - assert set(services) == expected, sorted(services) - assert all(item.get("running") for item in services.values()), services - assert all(item.get("health_status", {}).get("status") == "ok" - for name, item in services.items() - if name not in {"worker", "notifier"}), services - - gateway = services["gateway"]["port_actual"] - api = services["api"]["port_actual"] - web = services["web"]["port_actual"] - assert get_json(gateway, "/health")["ok"] is True - assert get_json(api, "/version")["version"] == "v1" - assert get_json(web, "/version")["version"].startswith("v1-") - - effects = { - "worker": root / ".berth" / state["worktree"]["identity"] / "worker.heartbeat", - "notifier": root / ".berth" / state["worktree"]["identity"] / "notifier.heartbeat", - } - for name, path in effects.items(): - deadline = time.monotonic() + 15 - while time.monotonic() < deadline: - if path.exists() and path.read_text().strip(): - last = json.loads([line for line in path.read_text().splitlines() if line.strip()][-1]) - if last.get("ok"): - break - time.sleep(.2) - else: - raise RuntimeError(f"{name} never wrote an ok heartbeat at {path}") - assert last["service"] == name, (last, name) - - trial.command(root, "down", "--force", "--json") - stopped = trial.status(root) - assert not any(item["running"] for item in stopped["worktree"]["services"]), stopped - return {"example": "stack-workshop", "services": len(services), - "ordered_dependencies": True, "slow_dependency": True, - "prepared_frontend": True, "migration": True, - "portless_workers": 2, "down_stops_all": "passed"} - finally: - trial.close() - - -def compose_available() -> bool: - try: - done = subprocess.run( - ["docker", "compose", "version"], - capture_output=True, text=True, timeout=15, - ) - except (OSError, subprocess.SubprocessError): - return False - return done.returncode == 0 - - -def compose(root: Path, project: str, *args: str, timeout: float = 120) -> subprocess.CompletedProcess[str]: - return subprocess.run( - ["docker", "compose", "-f", str(root / "docker-compose.yml"), "-p", project, *args], - cwd=root, capture_output=True, text=True, timeout=timeout, - ) - - -def compose_containers(project: str) -> list[str]: - # Running only. `down` stops the containers; it does not remove them, so - # `docker ps -a` would still list stopped ones and read as "left running". - done = subprocess.run( - ["docker", "ps", "-q", "--filter", f"label=com.docker.compose.project={project}"], - capture_output=True, text=True, timeout=30, - ) - if done.returncode: - # An unavailable daemon must not read as "no containers": that would - # make the after-down assertion pass without observing anything. - raise RuntimeError(f"docker ps for {project}: {done.stderr.strip()}") - return [line for line in done.stdout.splitlines() if line.strip()] - - -def run_oberth_json(trial, root: Path, *args: str, timeout: float = 120) -> dict: - done = subprocess.run( - [str(trial.binary), *args], cwd=root, env=trial.env, - capture_output=True, text=True, timeout=timeout, - ) - if done.returncode: - raise RuntimeError( - f"{args}: exit={done.returncode}\n{done.stderr}\n{done.stdout}" - ) - return json.loads(done.stdout) - - -def verify_compose(module, binary: Path, example: Path) -> dict: - """One Compose project per worktree, verified where Docker exists. - - macOS runners have no Docker daemon, so this branch reports a skip instead - of pretending the stack ran. The required Examples check still runs both - operating systems; the Compose evidence is the Linux row. - """ - if not sys.platform.startswith("linux"): - return {"example": "compose-stack", "skipped": "Docker Compose is verified on Linux CI"} - if not compose_available(): - # On the Linux row this is a missing prerequisite, not a platform - # difference: fail so the required check cannot pass without evidence. - raise RuntimeError("docker compose is not available on the Linux examples runner") - - # Pull first: `up --wait` must not spend its budget on an image download. - pulled = subprocess.run( - ["docker", "pull", "postgres:16-alpine"], - capture_output=True, text=True, timeout=300, - ) - if pulled.returncode: - raise RuntimeError("postgres:16-alpine could not be pulled: " + pulled.stderr.strip()) - - with tempfile.TemporaryDirectory(prefix="bcompose-", dir="/tmp") as tmp: - trial = module.Trial(binary, Path(tmp).resolve()) - projects: list[str] = [] - try: - trial.provision() - for root in trial.roots: - for name in ("docker-compose.yml", "oberth.yaml"): - shutil.copyfile(example / name, root / name) - - identities = [trial.status(root)["worktree"]["identity"] for root in trial.roots] - projects = ["app_" + identity for identity in identities] - for root, project in zip(trial.roots, projects): - run_oberth_json(trial, root, "up", "--wait", "--json") - - ports = [] - for root, project, identity in zip(trial.roots, projects, identities): - state = trial.status(root) - db = next(item for item in state["worktree"]["services"] if item["name"] == "db") - assert db["running"], db - deadline = time.monotonic() + 60 - while time.monotonic() < deadline: - health = db.get("health_status") or {} - if health.get("status") == "ok": - break - time.sleep(.2) - state = trial.status(root) - db = next(item for item in state["worktree"]["services"] if item["name"] == "db") - assert (db.get("health_status") or {}).get("status") == "ok", db - assert db.get("port_actual"), db - ports.append(db["port_actual"]) - - mapped = compose(root, project, "port", "db", "5432") - if mapped.returncode: - raise RuntimeError(f"compose port: {mapped.stderr}") - host_port = int(mapped.stdout.strip().rsplit(":", 1)[1]) - assert host_port == db["port_actual"], (host_port, db["port_actual"]) - - database = f"app_{identity}" - query = compose( - root, project, "exec", "-T", "db", "psql", "-U", "app", - "-d", database, "-tAc", "select current_database()", - ) - if query.returncode: - raise RuntimeError(f"psql in {project}: {query.stderr}") - assert query.stdout.strip() == database, (query.stdout, database) - assert compose_containers(project), f"no containers for {project}" - - assert ports[0] != ports[1], f"worktrees share a Compose host port: {ports}" - - # A normal down is SIGTERM: Compose must stop its own containers and - # leave the other worktree's project alone. - run_oberth_json(trial, trial.roots[0], "down", "--json") - deadline = time.monotonic() + 60 - while time.monotonic() < deadline and compose_containers(projects[0]): - time.sleep(.2) - assert not compose_containers(projects[0]), f"down left {projects[0]} running" - assert compose_containers(projects[1]), "down stopped the other worktree's containers" - other = trial.status(trial.roots[1]) - other_db = next(item for item in other["worktree"]["services"] if item["name"] == "db") - assert other_db["running"], other_db - assert (other_db.get("health_status") or {}).get("status") == "ok", other_db - - run_oberth_json(trial, trial.roots[1], "down", "--json") - deadline = time.monotonic() + 60 - while time.monotonic() < deadline and compose_containers(projects[1]): - time.sleep(.2) - assert not compose_containers(projects[1]), f"down left {projects[1]} running" - - return {"example": "compose-stack", "worktrees": 2, - "distinct_ports": True, "distinct_databases": True, - "port_auto_into_compose": True, "pg_isready_health": True, - "single_side_stop": "passed", "down_stops_containers": True} - finally: - for root, project in zip(trial.roots, projects): - try: - compose(root, project, "down", "--volumes", "--remove-orphans") - except (OSError, subprocess.SubprocessError): - pass - trial.close() - - -def verify(binary: Path) -> None: - module = trial_module() - examples = Path(__file__).resolve().parent.parent / "examples" - checked = [verify_hello(module, binary.resolve(strict=True), examples / "hello-worktree"), - verify_worktree_data(module, binary, examples / "worktree-data"), - verify_stack(module, binary, examples / "stack-workshop"), - verify_compose(module, binary, examples / "compose-stack")] - for result in checked: - print(json.dumps(result)) - print("isolated example processes and temporary files cleaned up") - - -if __name__ == "__main__": - parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("--binary", type=Path, required=True) - verify(parser.parse_args().binary) From 4b59749b2117f10e17859246e300ce212ebd0824 Mon Sep 17 00:00:00 2001 From: zayJu Date: Thu, 12 Nov 2026 11:45:16 +0800 Subject: [PATCH 4/5] =?UTF-8?q?docs:=20=E7=BB=9F=E4=B8=80=E7=A4=BA?= =?UTF-8?q?=E4=BE=8B=E3=80=81=E5=86=92=E7=83=9F=E4=B8=8E=E5=9F=BA=E5=87=86?= =?UTF-8?q?=E7=9A=84=E8=AF=B4=E6=B3=95?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 对外只用三个词：示例项目（sample）、冒烟（smoke）、基准（benchmark）。不再把 fixture / scenario / suite / tier / arm 混着用；基准代码里的 scenario/suite/arm 在 agent-eval.md 开头给出一次对应关系。 同步事实：Examples 每 PR 只跑 hello-worktree 冒烟，不需要 Docker；stack-workshop 是手动基准的输入，需要 Linux + Docker；cli.md 的 ${worktree} 指针仍指向它；validation 与 backlog 的平台范围和缺失项一起改。 验证：python3 scripts/check_docs.py 6 对双语文档一致。 Co-Authored-By: Claude Code --- README.md | 7 ++++--- README.zh.md | 6 +++--- docs/agent/agent-eval.md | 39 ++++++++++++++++++++++----------------- docs/agent/develop.md | 6 +++--- docs/agent/validation.md | 34 +++++++++++++++++++--------------- docs/backlog.md | 4 ++-- docs/cli.md | 8 ++++---- docs/cli.zh.md | 6 +++--- 8 files changed, 60 insertions(+), 50 deletions(-) diff --git a/README.md b/README.md index a9d28744..0a015d9c 100644 --- a/README.md +++ b/README.md @@ -43,9 +43,10 @@ lifecycle boundaries. The repository ships [Hello Worktree](examples/hello-worktree/README.md): one API that only listens on localhost and one portless worker, Python 3 standard library only — no model account, container or pip -dependency. [Worktree Data](examples/worktree-data/README.md) adds a database per worktree through -`${worktree}`, and [Stack Workshop](examples/stack-workshop/README.md) is a ten-service stack with a -slow-starting auth, a built frontend, a migration and read-only `machine:` dependencies. +dependency. [Stack Workshop](examples/stack-workshop/README.md) is the complex sample, and the input of +the manual agent benchmark: ten Python services plus a real Postgres container, with a slow-starting +auth, a built frontend, a migration per worktree, portless workers and a read-only `machine:` +reference. It needs Docker. ```sh # With oberth installed, from the repository root diff --git a/README.zh.md b/README.zh.md index 60e1874f..b0ad39de 100644 --- a/README.zh.md +++ b/README.zh.md @@ -41,9 +41,9 @@ checkout 应该跑什么；引擎在边界内启动、观察和停止；人和 c 仓库自带 [Hello Worktree](examples/hello-worktree/README.md)：一个只监听本机的 API 和一个无端口 worker，只需 Python 3 标准库，不需要模型账号、容器或 pip 依赖。 -[Worktree Data](examples/worktree-data/README.md) 用 `${worktree}` 给每份 worktree 各自的数据库； -[Stack Workshop](examples/stack-workshop/README.md) 是一份十服务整栈：慢启动的 auth、要构建的前端、 -要迁移的数据库，以及只读的 `machine:` 依赖。 +[Stack Workshop](examples/stack-workshop/README.md) 是复杂示例，也是手动 agent 基准的输入： +十个 Python 服务加一个真实的 Postgres 容器，包含慢启动的 auth、要构建的前端、每 worktree 一份的 +迁移、无端口 worker，以及只读的 `machine:` 引用。它需要 Docker。 ```sh # 已安装 oberth 后，从仓库根目录开始 diff --git a/docs/agent/agent-eval.md b/docs/agent/agent-eval.md index 62ecba26..8817f201 100644 --- a/docs/agent/agent-eval.md +++ b/docs/agent/agent-eval.md @@ -1,20 +1,25 @@ -# Agent scenario library and release matrix +# Agent benchmark -The entry point is `scripts/agent_eval.py`. It turns the manual acceptance round of 2026-10-28 into a -fixed scenario library, reruns it on the **same build** every time, and outputs completion rate, step -count, duration, output bytes, process probes, wrongful stops and boundary-violation counts for -"scenario × agent × control arm". +`scripts/agent_eval.py` is the manual benchmark. It asks an agent to perform one task on a sample +project, with and without oberth, reruns the same build every time, and reports completion rate, step +count, duration, output bytes, process probes, wrongful stops and boundary violations. -Scenarios come in three tiers: +Vocabulary: a **task** is one thing the agent is asked to do; a **group** is a set of tasks (the +`--suite` value); a **mode** is `oberth` or `native` (the `--arms` value). The code and `--scenarios` +call a task a "scenario"; that is the same thing under an older name. + +Tasks come in three groups: - **core** (`scripts/agent_eval_scenarios.py`): single-file small services, each proving one failure path. -- **large** (`scripts/agent_eval_stacks.py`): a ten-service stack covering slow dependencies, build - artifacts, database migration, a worker without a port, and three worktrees in parallel. The service - scripts are shared with the public example +- **large** (`scripts/agent_eval_stacks.py`): ten Python services plus a real Postgres container, + covering slow dependencies, build artifacts, database migration, a worker without a port, three + worktrees in parallel and the container lifecycle. The service scripts and + `docker-compose.yml` are shared with the sample [`examples/stack-workshop`](../../examples/stack-workshop/README.md); no second copy is maintained. + This group needs Docker and runs on Linux. - **cold** (`scripts/agent_eval_cold.py`): a repository with **no** `oberth.yaml`, so the run has to - draft, adopt and then start the project. The other two tiers begin from a manifest a person already + draft, adopt and then start the project. The other two groups begin from a manifest a person already wrote; this one measures the step a new user actually takes. Judgement uses facts published by the engine, loopback HTTP, ports, effect files and the PIDs of @@ -541,13 +546,13 @@ reference-agent or fake-engine rows. ## Platform and boundaries -- Supports macOS / Linux. The shim uses a Unix shebang; Windows is out of scope for this tool. -- The large-project fixtures likewise use only the Python 3 standard library; there are no Docker, JVM - or Node dependencies. Real JVM / Node builds / compose Postgres+Redis are a separate manual tier and - do not enter the standing matrix. -- The large-project fixtures start ten to thirty real processes and must run in an environment that can - observe listeners; in a sandbox `--dry-run --suite large` exits 2 at preflight or at the first real - start — this is an environment limitation, not a fixture failure. +- The harness supports macOS / Linux; the large group needs Docker and runs on Linux (the workflow uses + `ubuntu-24.04`). The shim uses a Unix shebang; Windows is out of scope for this tool. +- The large group starts ten to thirty Python processes plus one Postgres container per worktree and + must run in an environment that can observe listeners. When Docker is missing, `--dry-run --suite + large` exits 2 at preflight; that is an environment limit, not a fixture failure. +- JVM and Node builds are not part of the benchmark; the Python services plus Postgres are the only + runtimes it measures. - Provider event parsing is implemented against known shapes; when a shape changes, step count degrades to command count and is annotated, never silently zeroed. - Timeouts, authentication failures and provider crashes are recorded as `agent_error` and must not be diff --git a/docs/agent/develop.md b/docs/agent/develop.md index 0e4e8a72..826e2cfb 100644 --- a/docs/agent/develop.md +++ b/docs/agent/develop.md @@ -73,9 +73,9 @@ Scripts, release and brand generation: python3 -m unittest discover -s scripts -p '*_test.py' python3 scripts/brand_export.py --check python3 scripts/check_skill_contract.py --skill-dir ../option-berth-skill -python3 scripts/agent_eval.py --list # core/large/cold scenarios and the comparison arms -python3 scripts/agent_eval.py --dry-run --suite large # build the large fixture, launch no agent -python3 scripts/agent_eval.py --dry-run --suite cold # cold-start fixtures: no manifest, no agent +python3 scripts/agent_eval.py --list # core/large/cold groups and the comparison modes +python3 scripts/agent_eval.py --dry-run --suite large # build the complex samples; needs Linux + Docker +python3 scripts/agent_eval.py --dry-run --suite cold # cold-start samples: no manifest, no agent ``` `check_skill_contract.py` needs a sibling skill checkout; CI checks out `option-berth-skill` in the diff --git a/docs/agent/validation.md b/docs/agent/validation.md index 28652a44..36361d2b 100644 --- a/docs/agent/validation.md +++ b/docs/agent/validation.md @@ -19,7 +19,7 @@ workflow in Actions; to re-run locally before a release, use `python3 scripts/re | Engine verification | PRs, engine-related pushes to `main`, manual | Automated: macOS / Ubuntu native build, vet, full engine unit tests, Mage workflow regression, Python helper tests, cross builds for the six release targets on Ubuntu, a checkout of `option-berth-skill` on Linux to verify agent contract coverage, and parsing of the install/uninstall scripts with Windows PowerShell 5.1. Manual runs additionally execute core race and real-service scenarios. | | Nightly full verification | Daily 18:23 UTC, manual | macOS / Ubuntu: the same full gate as `scripts/release.py --verify-only`, including core race and real-service scenarios. | | Git read verification | PRs, related pushes to `main`, manual | macOS / Ubuntu: repeated tests and race for Git, worktree, attribution, and observation. | -| Checked-in examples | PRs, engine-related pushes to `main`, manual | macOS / Ubuntu: verify the checked-in examples across two worktrees, including the separate database, the idempotent migration, and stopping one side. Compose examples are verified only on Linux rows with Docker (separate project per worktree, `port: auto` mapping, `pg_isready`, and `down` on one side to stop the containers); macOS runners have no Docker, so the report shows a skip, which does not count as platform evidence. | +| Checked-in examples | PRs, engine-related pushes to `main`, manual | macOS / Ubuntu: the minimal sample `hello-worktree` across two worktrees (distinct ports, a portless worker, stopping one side). No Docker, no benchmark. The complex sample `stack-workshop` is not part of this check; it is the input of the manual benchmark, which runs on Linux with Docker. | | Reused service readiness | PRs, engine-related pushes to `main`, manual | macOS / Ubuntu: reuse checks against real healthy and unhealthy services. | | Windows build verification | PRs, engine-related pushes to `main`, release tags, manual | Windows x64: native build, vet, CLI/Jev build, version JSON/help page, facts/persistence, and start, repeated start, collector recovery, and scoped stop for an isolated API / portless worker; Ubuntu: Windows arm64 cross build. | | Gate integrity | PR (`pull_request_target`) | Requires the PR author to be a repository maintainer when `.github/workflows/`, CODEOWNERS, the race list, release scripts, or quality-gate check scripts are modified. | @@ -512,12 +512,12 @@ through `scripts/agent_eval.py`; the contract and metrics are in [agent-eval.md] - **large**: full-stack start of the ten-service stack, starting only a sub-stack, slow dependency, old code still running, three worktrees in parallel; plus three accuracy-lure scenarios (the port belongs to an external process, a same-name service in another checkout, only the frontend - changed). The fixtures are shared with the public example [stack-workshop](../../examples/stack-workshop/README.md). + changed). The sample and its scripts are shared with [stack-workshop](../../examples/stack-workshop/README.md). Each run uses an isolated `HOME` / `BERTH_HOME` / disposable project, and passes credentials and provider parameters only through the named `--pass-env` and `--agent-arg`; the harness itself adds -no approval-bypass or sandbox-disabling arguments. The `oberth` and `native` control arms use the -same fixtures and the same verdicts, the native arm does not stage the skill, and the `oberth` shim +no approval-bypass or sandbox-disabling arguments. The `oberth` and `native` modes use the same +sample and the same verdicts, the native mode does not stage the skill, and the `oberth` shim exits with 127. The report gives completion rate, step count, `oberth` calls, process probes, output bytes, elapsed @@ -631,10 +631,12 @@ was actually run on this machine; the CI matrix below the change is not implied. The baseline rerun that would produce that comparison was triggered on `main` but had not been approved when this was written. -## Manual tier: real JVM / Node / Compose (2026-11-01) +## Manual runs: real JVM / Node / Compose (2026-11-01) -The large-stack fixtures use only the Python standard library on purpose. These three runs check -that the engine also drives real toolchains; they are manual, not part of the standing matrix. +The complex sample is Python plus one Postgres container. These three runs checked that the engine also +drives real toolchains; they were manual, not part of the benchmark matrix. The Compose part has since +become the `db` service of the [stack-workshop](../../examples/stack-workshop/README.md) sample, which +the manual benchmark uses; it is not part of the per-PR Examples check. Environment: macOS 27.2 / 26B123, arm64; OpenJDK 21.0.12; Maven 3.9; Node 26.9.0; Docker 29.8.2 through colima, Compose 5.5.1. Every run used a throwaway `HOME` / `BERTH_HOME` under `/tmp` and a @@ -652,18 +654,20 @@ disposable git repository, and every daemon and container was removed at the end with `/health`; `dist/` is gitignored. `up --wait` reported `prepare 440ms`, `spawning 443ms`, `✓ web http://localhost:19670`. After editing `src/app.html`, `status` reported `code changed since start (1 file)`; `down` + `up --wait` rebuilt and served the new content. -- **Compose**: the checked-in [compose-stack](../../examples/compose-stack/README.md) example - (`postgres:16-alpine`, `port: auto`, `pg_isready` health command). Two environment preconditions - had to be met before it could run, and both are properties of this machine rather than of the - example: the isolated `HOME` hid `~/.docker/cli-plugins`, so `docker compose` degraded to - `docker` + an unknown subcommand (exit 125); and the colima context needs +- **Compose**: the standalone `compose-stack` example of this run (`postgres:16-alpine`, `port: auto`, + `pg_isready` health command), which has since become the `db` service of the checked-in + [stack-workshop](../../examples/stack-workshop/README.md) fixture. Two environment preconditions had + to be met before it could run, and both are properties of this machine rather than of the example: + the isolated `HOME` hid `~/.docker/cli-plugins`, so `docker compose` degraded to `docker` + an + unknown subcommand (exit 125); and the colima context needs `DOCKER_HOST=unix://$HOME/.colima/default/docker.sock`, without which the container client cannot reach the API. With both handled, the container started and `compose exec -T db pg_isready` answered `accepting connections`, but `oberth up --wait` still reported `timed out after 30s waiting for db to be ready` while `status` showed `auto port unassigned` - and the container was healthy. **This one is an open gap**: the compose example does not reach - ready on this machine, and the cause has not been isolated (the health command and the container - both work when driven by hand). It is recorded as a failure, not as a pass. + and the container was healthy. **This one is an open gap on this machine**: the standalone example + reached ready in the Linux Examples job, but this colima host did not, and the cause has not been + isolated (the health command and the container both work when driven by hand). It is recorded as a + failure, not as a pass. - **Not run**: the `native` arm on any of these three, and any real coding agent on them. They prove the engine drives real toolchains; they say nothing about agent behaviour. diff --git a/docs/backlog.md b/docs/backlog.md index 9bdd8760..f7ba6a9a 100644 --- a/docs/backlog.md +++ b/docs/backlog.md @@ -12,8 +12,8 @@ repository: the macOS client is | Priority | Capability | Current gap and completion condition | | --- | --- | --- | -| High | Whether the agent-facing capability surface is sufficient | The through-line is what a coding agent can do, so "can an agent take this code, run it itself, and judge for itself why it did not run" outranks any interface. **One round ran on 2026-10-28**: real Claude Code, on an unfamiliar worktree, using only the CLI and the skill, completed "start the service, fail, read the log, fix, start again" — the core loop in 6 steps (inputs and results in [validation.md](agent/validation.md) "Core acceptance"). The two places that round got stuck are fixed: a failure hint no longer points at a log that does not exist, and an unlistened `machine:` dependency no longer publishes a handoff (contract in [cli.md](cli.md)). **From 2026-10-30** that acceptance round was distilled into a fixed scenario library (missing artifact, port occupied, dependency not ready, manifest/process mismatch, same-repository dual worktree), see [agent-eval.md](agent/agent-eval.md). **From 2026-10-31** a ten-service large-project tier was added (slow dependency, generated artifact, migration, portless worker, three worktrees) plus `oberth` / `native` comparison arms, with the report outputting completion rate, steps, elapsed time, output bytes, process probing, wrong stops and cross-worktree differences; the fixture shares the public example [stack-workshop](../examples/stack-workshop/README.md). Still missing: a comparable release matrix from real Codex / OpenCode / Pi on an ordinary terminal that can observe listeners or in CI, plus one manual fixture tier with a real JVM / Node build / compose Postgres+Redis; preflight inside this machine's agent sandbox refuses to produce a table. | -| High | The evidence base for a release claim | Only half of #1 has been measured. The last real matrix (2026-11-01, CI run 41280627281, `main` at `81bf405`) covered **one** provider (Claude Code), the eight `large` scenarios, both arms and **repeat 2**; every row was comparable, but the table is a median over two runs with no spread, and nothing at all covers `core`, the four-provider claim, or `n >= 5`. The 2026-10-28 core acceptance is a single manual round. Completion condition: a matrix of `4 providers × (core + large) × n ≥ 5` with intervals in the report, plus the manual JVM / Node / compose tier above; until then no capability claim outranks the rows that actually exist. Two prerequisites are open work, not this row: the suite and arm generalization (a `cold` suite, an `oberth-noskill` arm, `--repeat` accepted as a range) does not exist yet, and `docs/agent/agent-eval.md` documents only `oberth` / `native`. | +| High | Whether the agent-facing capability surface is sufficient | The through-line is what a coding agent can do, so "can an agent take this code, run it itself, and judge for itself why it did not run" outranks any interface. **One round ran on 2026-10-28**: real Claude Code, on an unfamiliar worktree, using only the CLI and the skill, completed "start the service, fail, read the log, fix, start again" — the core loop in 6 steps (inputs and results in [validation.md](agent/validation.md) "Core acceptance"). The two places that round got stuck are fixed: a failure hint no longer points at a log that does not exist, and an unlistened `machine:` dependency no longer publishes a handoff (contract in [cli.md](cli.md)). **From 2026-10-30** that acceptance round was distilled into a fixed scenario library (missing artifact, port occupied, dependency not ready, manifest/process mismatch, same-repository dual worktree), see [agent-eval.md](agent/agent-eval.md). **From 2026-10-31** a ten-service large-project group was added (slow dependency, generated artifact, migration, portless worker, three worktrees) plus `oberth` / `native` comparison modes, with the report outputting completion rate, steps, elapsed time, output bytes, process probing, wrong stops and cross-worktree differences; the sample shares [stack-workshop](../examples/stack-workshop/README.md). Still missing: a comparable release matrix from real Codex / OpenCode / Pi on an ordinary terminal that can observe listeners or in CI, plus manual runs with a real JVM / Node build; compose Postgres now lives in the benchmark's large group. Preflight inside this machine's agent sandbox refuses to produce a table. | +| High | The evidence base for a release claim | Only half of #1 has been measured. The last real matrix (2026-11-01, CI run 41280627281, `main` at `81bf405`) covered **one** provider (Claude Code), the eight `large` scenarios, both arms and **repeat 2**; every row was comparable, but the table is a median over two runs with no spread, and nothing at all covers `core`, the four-provider claim, or `n >= 5`. The 2026-10-28 core acceptance is a single manual round. Completion condition: a matrix of `4 providers × (core + large) × n ≥ 5` with intervals in the report, plus the manual JVM / Node runs above; until then no capability claim outranks the rows that actually exist. Two prerequisites are open work, not this row: the suite and arm generalization (a `cold` suite, an `oberth-noskill` arm, `--repeat` accepted as a range) does not exist yet, and `docs/agent/agent-eval.md` documents only `oberth` / `native`. | | Medium | Cold-start baseline (a repository with no manifest) | Every scenario library row starts from a hand-written `oberth.yaml`; the step before it — an unfamiliar repository with no manifest at all, through draft → adopt → `up --wait` to a healthy service — is unmeasured, so there is no baseline against which a drafting improvement could be judged. Completion condition: a fixture suite with three repository shapes (compose + Makefile, Node monorepo, Maven multi-module) driven only by the Python standard library, reporting success rate, steps, elapsed time, how many times the manifest was edited after the first failure, undeclared and over-declared service counts, and the safety check that no tracked file other than `oberth.yaml` changed. `init draft` starts a second agent process, so each cold run costs one more model call and the budget belongs in the scenario. Do not change drafting (`internal/draft/brief.go`) before that baseline exists. | | Medium | End-to-end verification of orchestrator recipes | The Conductor / Cursor / Claude Code / Codex configurations in [integrations.md](integrations.md) come from each tool's official documentation and have not been run end to end in the real tools for creation, start and archive/cleanup. Completion condition: run setup, `status --json` and cleanup once each in an isolated `HOME` and a throwaway repository (including the path where the checkout was already deleted), recording the actual inputs and results. | diff --git a/docs/cli.md b/docs/cli.md index 6640e86d..0d8c2abc 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -849,10 +849,10 @@ engine only reports the correspondence between identities and paths. Only the five shapes in the table above are references; other `${...}` (for example `${HOME}`, `${FOO:-x}`) are kept as-is. `${worktree}` does not point at a service, so it neither needs nor produces `depends_on`; undeclared services and port references to a service with no port are still -rejected by manifest validation. For a runnable example see -[`examples/worktree-data`](../examples/worktree-data/README.md); for the per-worktree Docker Compose -style see [`examples/compose-stack`](../examples/compose-stack/README.md), which validates -`port: auto`, the `pg_isready` health check and `down` really stopping containers on Linux CI. +rejected by manifest validation. For a runnable sample see +[`examples/stack-workshop`](../examples/stack-workshop/README.md): it puts `${worktree}` into both a +SQLite path and a per-worktree Compose project, and its README shows `port: auto` reaching Compose as +`PORT`, the `pg_isready` health check and `down` letting Compose stop its containers. ### Shared environment files diff --git a/docs/cli.zh.md b/docs/cli.zh.md index 45e187da..558303e0 100644 --- a/docs/cli.zh.md +++ b/docs/cli.zh.md @@ -717,9 +717,9 @@ issue 上可见；否则命令打印一条预填好的 `issues/new` 链接由人 只有上表五种形状是引用；其他 `${...}`（例如 `${HOME}`、`${FOO:-x}`）原样保留。 `${worktree}` 不指向服务，不需要也不产生 `depends_on`；未声明的服务、没有端口的端口引用 -仍按清单校验拒绝。可运行示例见 [`examples/worktree-data`](../examples/worktree-data/README.md)； -Docker Compose 的每 worktree 一套写法见 [`examples/compose-stack`](../examples/compose-stack/README.md)， -它在 Linux CI 上验证 `port: auto`、`pg_isready` 健康检查和 `down` 真正停止容器。 +仍按清单校验拒绝。可运行示例见 [`examples/stack-workshop`](../examples/stack-workshop/README.md)： +它把 `${worktree}` 同时用于 SQLite 路径和每 worktree 一套的 Compose 项目；README 里能看到 +`port: auto` 以 `PORT` 进入 Compose、`pg_isready` 健康检查，以及 `down` 让 Compose 自己停容器。 ### 共享环境文件 From 9c13a39c405da011bc11c1f18cb4fe705cc14e06 Mon Sep 17 00:00:00 2001 From: zayJu Date: Thu, 12 Nov 2026 11:47:55 +0800 Subject: [PATCH 5/5] =?UTF-8?q?fix(benchmark):=20=E6=B2=A1=E6=9C=89=20comp?= =?UTF-8?q?ose=20=E5=A4=B9=E5=85=B7=E7=9A=84=20run=20=E4=B8=8D=E5=81=9A?= =?UTF-8?q?=E5=AE=B9=E5=99=A8=E6=B8=85=E7=90=86?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit core/cold 组没有 docker-compose.yml，清理时不需要探一次 Docker daemon；避免 daemon 未运行时给每次 run 留一条无意义的清理告警。 Co-Authored-By: Claude Code --- scripts/agent_eval_lib.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/scripts/agent_eval_lib.py b/scripts/agent_eval_lib.py index e3b894c6..f026138f 100644 --- a/scripts/agent_eval_lib.py +++ b/scripts/agent_eval_lib.py @@ -829,6 +829,8 @@ def _cleanup_containers(self) -> list[str]: container survives that stop. Compose labels carry the project and its working directory; anything under this run's root belongs to the run. """ + if not any((root / "docker-compose.yml").exists() for root in self.projects): + return [] try: listed = subprocess.run( ["docker", "ps", "-a", "--format",