Found while verifying test coverage for the remote-only epic (#118).
The gap
`pyproject.toml` sets `addopts = "-m 'not e2e and not live'"`, so `pytest -q`
(what CI runs) never executes the 11 tests marked `e2e`/`live`:
- `tests/examples/test_bench_e2e.py` (5 tests) — real FastAPI `TestClient` server,
real `Governor`, `step_cap`/`cost_budget`/`tool_output_cap` policies actually
HALTing/MUTATing. No API key needed — model + search tool are faked.
- `tests/examples/test_triad_e2e.py` (5 tests) — real 3-agent (planner→researcher→writer)
A2A pipeline, shared ledger, per-agent `step_cap`, `cost_budget` halting one agent.
Also no API key needed.
- `tests/benchmarking/test_metagpt_live_scenarios.py` (1 test) — genuinely needs a live
API key; correctly self-skips without one.
These are the repo's only tests that exercise real multi-step policy enforcement
(actual HALT/MUTATE decisions from real policies, not `FakeView`-pinned unit tests) and
a real multi-agent pipeline with a shared ledger. They're valuable and none of the
10 non-live ones need network access — there's no reason they can't run in CI.
3 are currently broken (pre-existing on `main`, unrelated to #118 work)
Running them explicitly (`pytest -m e2e`) on current `main`:
- `test_triad_e2e.py::test_triad_pipeline_completes_with_ledger` and
`::test_triad_cost_not_double_counted_without_parent_rollup` —
`AttributeError: <module 'examples.triad.planner.server'> has no attribute
'build_price_book'`. The test does
`monkeypatch.setattr(planner_srv, "build_price_book", unit_price)`, but
`build_price_book` is only imported at module scope in `tokenops/control/run.py`
(`tokenops_run` calls it when `price=None`) — the triad server modules never import
it into their own namespace, so there's nothing there to patch. Fix: patch
`tokenops.control.run.build_price_book` instead (or pass `price=` explicitly through
`tokenops_run`).
- `test_bench_e2e.py::test_run_dims_persisted_for_segmentation` — real assertion
failure: a custom `user_dims` key (`"team": "growth"`) never reaches
`RunRecord.dims`; only `user_id`/`user` do. Matches
`attribution._PAYLOAD_USER_DIM_ALLOWLIST = frozenset({"user_id", "user"})` — either
the test is stale (allow-list is intentional and narrower than the test assumes) or
the allow-list should include a way to pass arbitrary segmentation tags. Needs a
product decision, not just a test fix.
Suggested fix
- Add an `e2e` CI job (or a `make e2e` target) that runs `pytest -m e2e` — the 10
non-live ones need no secrets, so this is close to free.
- Fix the two `build_price_book` monkeypatch targets.
- Decide the `test_run_dims_persisted_for_segmentation` question (widen the
allow-list vs. fix the test's expectation) and fix whichever side is wrong.
Related: #118 (remote-only epic) surfaced this while confirming end-to-end test
coverage; not caused by any of #119/#120/#121.
Found while verifying test coverage for the remote-only epic (#118).
The gap
`pyproject.toml` sets `addopts = "-m 'not e2e and not live'"`, so `pytest -q`
(what CI runs) never executes the 11 tests marked `e2e`/`live`:
real `Governor`, `step_cap`/`cost_budget`/`tool_output_cap` policies actually
HALTing/MUTATing. No API key needed — model + search tool are faked.
A2A pipeline, shared ledger, per-agent `step_cap`, `cost_budget` halting one agent.
Also no API key needed.
API key; correctly self-skips without one.
These are the repo's only tests that exercise real multi-step policy enforcement
(actual HALT/MUTATE decisions from real policies, not `FakeView`-pinned unit tests) and
a real multi-agent pipeline with a shared ledger. They're valuable and none of the
10 non-live ones need network access — there's no reason they can't run in CI.
3 are currently broken (pre-existing on `main`, unrelated to #118 work)
Running them explicitly (`pytest -m e2e`) on current `main`:
`::test_triad_cost_not_double_counted_without_parent_rollup` —
`AttributeError: <module 'examples.triad.planner.server'> has no attribute
'build_price_book'`. The test does
`monkeypatch.setattr(planner_srv, "build_price_book", unit_price)`, but
`build_price_book` is only imported at module scope in `tokenops/control/run.py`
(`tokenops_run` calls it when `price=None`) — the triad server modules never import
it into their own namespace, so there's nothing there to patch. Fix: patch
`tokenops.control.run.build_price_book` instead (or pass `price=` explicitly through
`tokenops_run`).
failure: a custom `user_dims` key (`"team": "growth"`) never reaches
`RunRecord.dims`; only `user_id`/`user` do. Matches
`attribution._PAYLOAD_USER_DIM_ALLOWLIST = frozenset({"user_id", "user"})` — either
the test is stale (allow-list is intentional and narrower than the test assumes) or
the allow-list should include a way to pass arbitrary segmentation tags. Needs a
product decision, not just a test fix.
Suggested fix
non-live ones need no secrets, so this is close to free.
allow-list vs. fix the test's expectation) and fix whichever side is wrong.
Related: #118 (remote-only epic) surfaced this while confirming end-to-end test
coverage; not caused by any of #119/#120/#121.