Skip to content

e2e/live tests excluded from CI, and 3 of 11 are currently broken on main #127

Description

@susheem-k

Found while verifying test coverage for the remote-only epic (#118).

The gap

`pyproject.toml` sets `addopts = "-m 'not e2e and not live'"`, so `pytest -q`
(what CI runs) never executes the 11 tests marked `e2e`/`live`:

  • `tests/examples/test_bench_e2e.py` (5 tests) — real FastAPI `TestClient` server,
    real `Governor`, `step_cap`/`cost_budget`/`tool_output_cap` policies actually
    HALTing/MUTATing. No API key needed — model + search tool are faked.
  • `tests/examples/test_triad_e2e.py` (5 tests) — real 3-agent (planner→researcher→writer)
    A2A pipeline, shared ledger, per-agent `step_cap`, `cost_budget` halting one agent.
    Also no API key needed.
  • `tests/benchmarking/test_metagpt_live_scenarios.py` (1 test) — genuinely needs a live
    API key; correctly self-skips without one.

These are the repo's only tests that exercise real multi-step policy enforcement
(actual HALT/MUTATE decisions from real policies, not `FakeView`-pinned unit tests) and
a real multi-agent pipeline with a shared ledger. They're valuable and none of the
10 non-live ones need network access — there's no reason they can't run in CI.

3 are currently broken (pre-existing on `main`, unrelated to #118 work)

Running them explicitly (`pytest -m e2e`) on current `main`:

  • `test_triad_e2e.py::test_triad_pipeline_completes_with_ledger` and
    `::test_triad_cost_not_double_counted_without_parent_rollup` —
    `AttributeError: <module 'examples.triad.planner.server'> has no attribute
    'build_price_book'`. The test does
    `monkeypatch.setattr(planner_srv, "build_price_book", unit_price)`, but
    `build_price_book` is only imported at module scope in `tokenops/control/run.py`
    (`tokenops_run` calls it when `price=None`) — the triad server modules never import
    it into their own namespace, so there's nothing there to patch. Fix: patch
    `tokenops.control.run.build_price_book` instead (or pass `price=` explicitly through
    `tokenops_run`).
  • `test_bench_e2e.py::test_run_dims_persisted_for_segmentation` — real assertion
    failure: a custom `user_dims` key (`"team": "growth"`) never reaches
    `RunRecord.dims`; only `user_id`/`user` do. Matches
    `attribution._PAYLOAD_USER_DIM_ALLOWLIST = frozenset({"user_id", "user"})` — either
    the test is stale (allow-list is intentional and narrower than the test assumes) or
    the allow-list should include a way to pass arbitrary segmentation tags. Needs a
    product decision, not just a test fix.

Suggested fix

  1. Add an `e2e` CI job (or a `make e2e` target) that runs `pytest -m e2e` — the 10
    non-live ones need no secrets, so this is close to free.
  2. Fix the two `build_price_book` monkeypatch targets.
  3. Decide the `test_run_dims_persisted_for_segmentation` question (widen the
    allow-list vs. fix the test's expectation) and fix whichever side is wrong.

Related: #118 (remote-only epic) surfaced this while confirming end-to-end test
coverage; not caused by any of #119/#120/#121.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions