Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions .github/workflows/examples.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ on:
paths:
- 'examples/**'
- 'engine/**'
- 'scripts/verify-example.py'
- 'scripts/smoke.py'
- '.github/workflows/examples.yml'
workflow_dispatch:
permissions:
Expand All @@ -33,11 +33,13 @@ jobs:
with:
go-version-file: engine/go.mod
cache-dependency-path: engine/go.sum
- name: Verify the checked-in examples in two worktrees
# The minimal sample only: no Docker, no benchmark. The complex sample
# runs in the manual benchmark, which is the only consumer that needs it.
- name: Run the per-PR smoke check
shell: bash
run: |
set -euo pipefail
output=$(mktemp -d)
trap 'rm -rf "$output"' EXIT
(cd engine && go build -o "$output/oberth" ./cmd/oberth)
python3 scripts/verify-example.py --binary "$output/oberth"
python3 scripts/smoke.py --binary "$output/oberth"
7 changes: 4 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,9 +43,10 @@ lifecycle boundaries.

The repository ships [Hello Worktree](examples/hello-worktree/README.md): one API that only listens on
localhost and one portless worker, Python 3 standard library only — no model account, container or pip
dependency. [Worktree Data](examples/worktree-data/README.md) adds a database per worktree through
`${worktree}`, and [Stack Workshop](examples/stack-workshop/README.md) is a ten-service stack with a
slow-starting auth, a built frontend, a migration and read-only `machine:` dependencies.
dependency. [Stack Workshop](examples/stack-workshop/README.md) is the complex sample, and the input of
the manual agent benchmark: ten Python services plus a real Postgres container, with a slow-starting
auth, a built frontend, a migration per worktree, portless workers and a read-only `machine:`
reference. It needs Docker.

```sh
# With oberth installed, from the repository root
Expand Down
6 changes: 3 additions & 3 deletions README.zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,9 +41,9 @@ checkout 应该跑什么;引擎在边界内启动、观察和停止;人和 c

仓库自带 [Hello Worktree](examples/hello-worktree/README.md):一个只监听本机的 API 和一个无端口
worker,只需 Python 3 标准库,不需要模型账号、容器或 pip 依赖。
[Worktree Data](examples/worktree-data/README.md) 用 `${worktree}` 给每份 worktree 各自的数据库;
[Stack Workshop](examples/stack-workshop/README.md) 是一份十服务整栈:慢启动的 auth、要构建的前端、
要迁移的数据库,以及只读的 `machine:` 依赖。
[Stack Workshop](examples/stack-workshop/README.md) 是复杂示例,也是手动 agent 基准的输入:
十个 Python 服务加一个真实的 Postgres 容器,包含慢启动的 auth、要构建的前端、每 worktree 一份的
迁移、无端口 worker,以及只读的 `machine:` 引用。它需要 Docker。

```sh
# 已安装 oberth 后,从仓库根目录开始
Expand Down
39 changes: 22 additions & 17 deletions docs/agent/agent-eval.md
Original file line number Diff line number Diff line change
@@ -1,20 +1,25 @@
# Agent scenario library and release matrix
# Agent benchmark

The entry point is `scripts/agent_eval.py`. It turns the manual acceptance round of 2026-10-28 into a
fixed scenario library, reruns it on the **same build** every time, and outputs completion rate, step
count, duration, output bytes, process probes, wrongful stops and boundary-violation counts for
"scenario × agent × control arm".
`scripts/agent_eval.py` is the manual benchmark. It asks an agent to perform one task on a sample
project, with and without oberth, reruns the same build every time, and reports completion rate, step
count, duration, output bytes, process probes, wrongful stops and boundary violations.

Scenarios come in three tiers:
Vocabulary: a **task** is one thing the agent is asked to do; a **group** is a set of tasks (the
`--suite` value); a **mode** is `oberth` or `native` (the `--arms` value). The code and `--scenarios`
call a task a "scenario"; that is the same thing under an older name.

Tasks come in three groups:

- **core** (`scripts/agent_eval_scenarios.py`): single-file small services, each proving one failure
path.
- **large** (`scripts/agent_eval_stacks.py`): a ten-service stack covering slow dependencies, build
artifacts, database migration, a worker without a port, and three worktrees in parallel. The service
scripts are shared with the public example
- **large** (`scripts/agent_eval_stacks.py`): ten Python services plus a real Postgres container,
covering slow dependencies, build artifacts, database migration, a worker without a port, three
worktrees in parallel and the container lifecycle. The service scripts and
`docker-compose.yml` are shared with the sample
[`examples/stack-workshop`](../../examples/stack-workshop/README.md); no second copy is maintained.
This group needs Docker and runs on Linux.
- **cold** (`scripts/agent_eval_cold.py`): a repository with **no** `oberth.yaml`, so the run has to
draft, adopt and then start the project. The other two tiers begin from a manifest a person already
draft, adopt and then start the project. The other two groups begin from a manifest a person already
wrote; this one measures the step a new user actually takes.

Judgement uses facts published by the engine, loopback HTTP, ports, effect files and the PIDs of
Expand Down Expand Up @@ -541,13 +546,13 @@ reference-agent or fake-engine rows.

## Platform and boundaries

- Supports macOS / Linux. The shim uses a Unix shebang; Windows is out of scope for this tool.
- The large-project fixtures likewise use only the Python 3 standard library; there are no Docker, JVM
or Node dependencies. Real JVM / Node builds / compose Postgres+Redis are a separate manual tier and
do not enter the standing matrix.
- The large-project fixtures start ten to thirty real processes and must run in an environment that can
observe listeners; in a sandbox `--dry-run --suite large` exits 2 at preflight or at the first real
start — this is an environment limitation, not a fixture failure.
- The harness supports macOS / Linux; the large group needs Docker and runs on Linux (the workflow uses
`ubuntu-24.04`). The shim uses a Unix shebang; Windows is out of scope for this tool.
- The large group starts ten to thirty Python processes plus one Postgres container per worktree and
must run in an environment that can observe listeners. When Docker is missing, `--dry-run --suite
large` exits 2 at preflight; that is an environment limit, not a fixture failure.
- JVM and Node builds are not part of the benchmark; the Python services plus Postgres are the only
runtimes it measures.
- Provider event parsing is implemented against known shapes; when a shape changes, step count degrades
to command count and is annotated, never silently zeroed.
- Timeouts, authentication failures and provider crashes are recorded as `agent_error` and must not be
Expand Down
6 changes: 3 additions & 3 deletions docs/agent/develop.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,9 +73,9 @@ Scripts, release and brand generation:
python3 -m unittest discover -s scripts -p '*_test.py'
python3 scripts/brand_export.py --check
python3 scripts/check_skill_contract.py --skill-dir ../option-berth-skill
python3 scripts/agent_eval.py --list # core/large/cold scenarios and the comparison arms
python3 scripts/agent_eval.py --dry-run --suite large # build the large fixture, launch no agent
python3 scripts/agent_eval.py --dry-run --suite cold # cold-start fixtures: no manifest, no agent
python3 scripts/agent_eval.py --list # core/large/cold groups and the comparison modes
python3 scripts/agent_eval.py --dry-run --suite large # build the complex samples; needs Linux + Docker
python3 scripts/agent_eval.py --dry-run --suite cold # cold-start samples: no manifest, no agent
```

`check_skill_contract.py` needs a sibling skill checkout; CI checks out `option-berth-skill` in the
Expand Down
34 changes: 19 additions & 15 deletions docs/agent/validation.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ workflow in Actions; to re-run locally before a release, use `python3 scripts/re
| Engine verification | PRs, engine-related pushes to `main`, manual | Automated: macOS / Ubuntu native build, vet, full engine unit tests, Mage workflow regression, Python helper tests, cross builds for the six release targets on Ubuntu, a checkout of `option-berth-skill` on Linux to verify agent contract coverage, and parsing of the install/uninstall scripts with Windows PowerShell 5.1. Manual runs additionally execute core race and real-service scenarios. |
| Nightly full verification | Daily 18:23 UTC, manual | macOS / Ubuntu: the same full gate as `scripts/release.py --verify-only`, including core race and real-service scenarios. |
| Git read verification | PRs, related pushes to `main`, manual | macOS / Ubuntu: repeated tests and race for Git, worktree, attribution, and observation. |
| Checked-in examples | PRs, engine-related pushes to `main`, manual | macOS / Ubuntu: verify the checked-in examples across two worktrees, including the separate database, the idempotent migration, and stopping one side. Compose examples are verified only on Linux rows with Docker (separate project per worktree, `port: auto` mapping, `pg_isready`, and `down` on one side to stop the containers); macOS runners have no Docker, so the report shows a skip, which does not count as platform evidence. |
| Checked-in examples | PRs, engine-related pushes to `main`, manual | macOS / Ubuntu: the minimal sample `hello-worktree` across two worktrees (distinct ports, a portless worker, stopping one side). No Docker, no benchmark. The complex sample `stack-workshop` is not part of this check; it is the input of the manual benchmark, which runs on Linux with Docker. |
| Reused service readiness | PRs, engine-related pushes to `main`, manual | macOS / Ubuntu: reuse checks against real healthy and unhealthy services. |
| Windows build verification | PRs, engine-related pushes to `main`, release tags, manual | Windows x64: native build, vet, CLI/Jev build, version JSON/help page, facts/persistence, and start, repeated start, collector recovery, and scoped stop for an isolated API / portless worker; Ubuntu: Windows arm64 cross build. |
| Gate integrity | PR (`pull_request_target`) | Requires the PR author to be a repository maintainer when `.github/workflows/`, CODEOWNERS, the race list, release scripts, or quality-gate check scripts are modified. |
Expand Down Expand Up @@ -512,12 +512,12 @@ through `scripts/agent_eval.py`; the contract and metrics are in [agent-eval.md]
- **large**: full-stack start of the ten-service stack, starting only a sub-stack, slow dependency,
old code still running, three worktrees in parallel; plus three accuracy-lure scenarios (the port
belongs to an external process, a same-name service in another checkout, only the frontend
changed). The fixtures are shared with the public example [stack-workshop](../../examples/stack-workshop/README.md).
changed). The sample and its scripts are shared with [stack-workshop](../../examples/stack-workshop/README.md).

Each run uses an isolated `HOME` / `BERTH_HOME` / disposable project, and passes credentials and
provider parameters only through the named `--pass-env` and `--agent-arg`; the harness itself adds
no approval-bypass or sandbox-disabling arguments. The `oberth` and `native` control arms use the
same fixtures and the same verdicts, the native arm does not stage the skill, and the `oberth` shim
no approval-bypass or sandbox-disabling arguments. The `oberth` and `native` modes use the same
sample and the same verdicts, the native mode does not stage the skill, and the `oberth` shim
exits with 127.

The report gives completion rate, step count, `oberth` calls, process probes, output bytes, elapsed
Expand Down Expand Up @@ -631,10 +631,12 @@ was actually run on this machine; the CI matrix below the change is not implied.
The baseline rerun that would produce that comparison was triggered on `main` but had not been
approved when this was written.

## Manual tier: real JVM / Node / Compose (2026-11-01)
## Manual runs: real JVM / Node / Compose (2026-11-01)

The large-stack fixtures use only the Python standard library on purpose. These three runs check
that the engine also drives real toolchains; they are manual, not part of the standing matrix.
The complex sample is Python plus one Postgres container. These three runs checked that the engine also
drives real toolchains; they were manual, not part of the benchmark matrix. The Compose part has since
become the `db` service of the [stack-workshop](../../examples/stack-workshop/README.md) sample, which
the manual benchmark uses; it is not part of the per-PR Examples check.

Environment: macOS 27.2 / 26B123, arm64; OpenJDK 21.0.12; Maven 3.9; Node 26.9.0; Docker 29.8.2
through colima, Compose 5.5.1. Every run used a throwaway `HOME` / `BERTH_HOME` under `/tmp` and a
Expand All @@ -652,18 +654,20 @@ disposable git repository, and every daemon and container was removed at the end
with `/health`; `dist/` is gitignored. `up --wait` reported `prepare 440ms`, `spawning 443ms`,
`✓ web http://localhost:19670`. After editing `src/app.html`, `status` reported
`code changed since start (1 file)`; `down` + `up --wait` rebuilt and served the new content.
- **Compose**: the checked-in [compose-stack](../../examples/compose-stack/README.md) example
(`postgres:16-alpine`, `port: auto`, `pg_isready` health command). Two environment preconditions
had to be met before it could run, and both are properties of this machine rather than of the
example: the isolated `HOME` hid `~/.docker/cli-plugins`, so `docker compose` degraded to
`docker` + an unknown subcommand (exit 125); and the colima context needs
- **Compose**: the standalone `compose-stack` example of this run (`postgres:16-alpine`, `port: auto`,
`pg_isready` health command), which has since become the `db` service of the checked-in
[stack-workshop](../../examples/stack-workshop/README.md) fixture. Two environment preconditions had
to be met before it could run, and both are properties of this machine rather than of the example:
the isolated `HOME` hid `~/.docker/cli-plugins`, so `docker compose` degraded to `docker` + an
unknown subcommand (exit 125); and the colima context needs
`DOCKER_HOST=unix://$HOME/.colima/default/docker.sock`, without which the container client cannot
reach the API. With both handled, the container started and `compose exec -T db pg_isready`
answered `accepting connections`, but `oberth up --wait` still reported
`timed out after 30s waiting for db to be ready` while `status` showed `auto port unassigned`
and the container was healthy. **This one is an open gap**: the compose example does not reach
ready on this machine, and the cause has not been isolated (the health command and the container
both work when driven by hand). It is recorded as a failure, not as a pass.
and the container was healthy. **This one is an open gap on this machine**: the standalone example
reached ready in the Linux Examples job, but this colima host did not, and the cause has not been
isolated (the health command and the container both work when driven by hand). It is recorded as a
failure, not as a pass.
- **Not run**: the `native` arm on any of these three, and any real coding agent on them. They prove
the engine drives real toolchains; they say nothing about agent behaviour.

Expand Down
Loading
Loading