Skip to content

0.2.0: ledger precheck + events:batch, run_state, data_scope (TokenOps remote-only) - #12

Merged
susheem-k merged 8 commits into
mainfrom
feat/ledger-precheck-events-batch
Sep 12, 2026
Merged

susheem-k merged 8 commits into
mainfrom
feat/ledger-precheck-events-batch

Conversation

@susheem-k

@susheem-k susheem-k commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Phase 1 of the TokenOps remote-only effort. Closes #10. Companion: theagentplane/tokenops#118.

Additive — nothing removed. A 0.1-era client keeps working against this. The
destructive fold (drop run_registrations, remove create_run, narrow PATCH) is
split out to #11 and ships as 0.3.0 after tokenops <next>not needed for the
decoupling.

What's here

POST /v1/ledger/events:batch batched writes: spent_add (multi-target fan-out), admit/complete, step, halt_mark/halt_clear; array order, one transaction, idempotent per idempotency_key (ledger_events table); ack returns totals + halted
POST /v1/ledger/precheck one read for a pre_call pass — halt + requested spent / inflight / window slices
run_state table per-run step_count / bounded window ring (64) / velocity, fed by step events
policy_instances.data_scope local|global (default local) — schema, migration ALTER, serde, PUT /v1/policies, YAML seed, emitted in GET /v1/governance/{agent}
POST /v1/runs returns the full record incl. registered_at — SDK binds without a follow-up GET .../registration
PATCH /v1/run-records client-sent steps / cost_micros are ignored + logged (derived server-side; 0.3.0 rejects)
GET|POST|DELETE /v1/ledger/runs/{run_id}/halt run-scoped halt; /v1/ledger/halt/* kept as aliases
GET /health advertises max_batch / max_body_bytes
PRAGMA user_version forward-only _apply_migrations() (SCHEMA_VERSION = 2); opening a 0.1 DB adds the new tables/column, bumps the version, no data loss. v3 = the #11 fold
docs/api-contract.md the ratified TokenOps ⇄ control-plane wire spec
test infra make_client fixture closes SQLite before temp-dir cleanup — the suite was fully red on Windows before this (open handles → WinError 32); test_health.py / test_envelopes.py migrated onto it

Tests

test_ledger_events.py (10), test_ledger_precheck.py (6), test_data_scope.py (5),
test_runs_and_halt.py (5), test_migrations.py (3, incl. a legacy user_version=0
DB upgrading additively). Full suite: 34 passed. ruff check clean.

Also in this PR: control-plane start / stop / status (Tier 1 one-click run)

Running the plane locally is now a prerequisite for TokenOps's remote-only work, so
this adds a managed background mode instead of requiring a dedicated terminal tab:

control-plane start    # detached, waits for /health before returning, prints pid + log path
control-plane status   # pid, port, version
control-plane stop     # terminate() -> kill() on timeout
  • Singleton per state dir (platformdirs user-state dir); all three are idempotent.
  • PID trust is verified (cmdline() match), not assumed — PIDs get reused by the OS.
  • Cross-platform detach: start_new_session (POSIX) / DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP (Windows).
  • start never lies: polls /health, kills and reports the log tail if the process never comes up.
  • Documented limitation: Windows has no SIGTERM, so stop there is a hard stop, not a graceful drain.
  • New deps: psutil, platformdirs.

tests/test_servicectl.py — real integration tests, no mocking of the OS layer: spawns
an actual subprocess, hits real HTTP. Full lifecycle (start → /health → status → stop
→ confirm the process actually exited), idempotent start/stop, stale-state-file
recovery, and a smoke pass over the real TokenOps-facing routes (/v1/runs,
/v1/governance, /v1/ledger/events:batch, /v1/ledger/precheck,
/v1/ledger/runs/{id}/halt) through the managed instance. The Windows detach/terminate
path was verified manually (this repo's CI is Ubuntu-only).

Full suite now: 42 passed. ruff check clean.

Not in this PR

🤖 Generated with Claude Code

susheem-k and others added 6 commits September 11, 2026 00:19
First increment of the 0.2.0 remote-only work (#10). Adds the batched ledger
write path the TokenOps SDK will use instead of per-op /v1/ledger/* calls:

- events:batch route (scope: ingest), Durability header validated, max_batch /
  max_body enforced.
- SqliteStore.apply_events(): applies spent_add (multi-target fan-out) / admit /
  complete / step / halt_mark / halt_clear in array order, one transaction,
  rolls back the whole batch on any bad event.
- ledger_events(tenant_id, idempotency_key) dedupe table — replayed keys are
  counted in `deduped`, not re-applied; ack still returns current totals.
- run_state(tenant_id, run_id) — step_count, bounded window ring (64), velocity;
  fed by `step` events; SqliteStore.get_run_state().
- docs/api-contract.md — the ratified TokenOps <-> control-plane wire contract.
- tests/test_ledger_events.py — fan-out, ordering, idempotent replay,
  admit/complete, step accumulation, halt, rollback, 400s.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…s on full dedup

- tests/conftest.py: make_client fixture closes SqliteStore/EnvelopeStore before
  removing the temp dir (open SQLite handles block unlink on Windows; the whole
  suite failed locally on Windows with WinError 32).
- migrate test_health.py / test_envelopes.py / test_ledger_events.py onto it.
- apply_events: collect spent_add targets into `touched` before the dedup check,
  so a fully-deduped batch still returns current totals in the ack (contract §6).

Full suite: 17 passed. ruff clean.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Consolidated pre_call read — one request returns halt state plus the requested
spent / inflight / window slices, replacing the 2-4 GETs the SDK does today.

- SqliteStore.precheck(tenant_id, run_id, segment_keys, budgets, want)
- route scope: read
- want defaults to [spent, inflight, halt]; "window" pulls the run_state slice
- unknown run -> zeros, not 404 (pre_call must not fail on a not-yet-seen run)

tests/test_ledger_precheck.py: 6 tests. Full suite 22 passed, ruff clean.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…dditive

Reframes 0.2.0 as additive-only on the wire and schema so a current tokenops
(API mode, no dep on this package) keeps working against it. The breaking fold
is deferred to control-plane 0.3.0, released after tokenops <next>.

- _migrate() -> _apply_migrations(): forward-only, tracked in PRAGMA user_version
  (SqliteStore.SCHEMA_VERSION = 2). Existing ad-hoc ALTERs become the legacy
  (version < 1) branch. Placeholder for the v3 destructive fold.
- _SCHEMA (CREATE IF NOT EXISTS) already adds ledger_events / run_state to any DB.
- docs/api-contract.md: §8/§13/§15 + header — 0.2.0 additive, run-records write
  path deprecated (kept, ignores steps/cost_micros), run_registrations drop +
  create_run removal -> 0.3.0.
- CHANGELOG.

tests/test_migrations.py: fresh DB at current version; reopen idempotent; a
legacy user_version=0 DB upgrades additively (new tables added, run_registrations
NOT dropped, data intact).

Full suite: 25 passed. ruff clean.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…TCH narrowing

Remaining additive 0.2.0 items (#10). Nothing removed — 0.1-era clients keep
working; the destructive fold stays scheduled for 0.3.0.

- policy_instances.data_scope (local|global, default local): models, schema,
  _ensure_columns ALTER for existing DBs, serde, upsert, YAML seed (params.pop),
  and emitted in governance_config_for output.
- POST /v1/runs returns registered_at; register_run returns the persisted row so
  the SDK binds without a follow-up GET .../registration. RunRegistration gains
  registered_at; _registration / serde read it.
- PATCH /v1/run-records: client-sent steps / cost_micros are dropped + logged
  (derived from run_state / ledger_spent; 0.3.0 will reject).
- GET|POST|DELETE /v1/ledger/runs/{run_id}/halt — run-scoped halt; the old
  /v1/ledger/halt/* routes stay as aliases.
- GET /health advertises max_batch / max_body_bytes.
- _migrate() -> _ensure_columns(): runs every open, fully guarded (fresh / 0.1 /
  intermediate DBs all converge).
- version -> 0.2.0 (pyproject, settings).
- docs: api-contract §8/§10/§13/§15, DESIGN.md TokenOps section, README
  compatibility + breaking-changes table, CHANGELOG 0.2.0.

tests: test_data_scope.py (5), test_runs_and_halt.py (5), test_migrations.py
extended (data_scope ALTER on a legacy DB). Full suite 34 passed, ruff clean.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
susheem-k and others added 2 commits September 11, 2026 13:19
Prerequisite for running tokenops locally against a real (non-mocked) plane:
one command to bring it up in the background, one to check it, one to take it
down -- no separate terminal tab to babysit.

- control_plane/servicectl.py: start()/stop()/status(), singleton per state dir
  (platformdirs user-state dir, overridable via CONTROL_PLANE_STATE_DIR for
  tests). start() spawns `python -m control_plane.cli serve` detached
  (POSIX: start_new_session; Windows: DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP),
  redirects stdout/stderr to a log file, and polls /health before declaring
  success -- a process that never comes up is reaped, not left orphaned and
  untracked. stop() escalates terminate() -> kill() on a timeout. Both are
  idempotent. PID checks confirm the process is actually a `control_plane.cli
  serve` invocation (cmdline match) before trusting a stored PID -- PIDs get
  reused.
  Explicit, undocumented-as-graceful limitation: Windows has no SIGTERM, so
  stop() there is a hard stop, not a drain.
- cli.py: `start` / `stop` / `status` subcommands.
- New deps: psutil, platformdirs.
- README quick start now leads with start/status/stop; serve documented as the
  foreground/scripting form.

tests/test_servicectl.py -- integration tests that spawn a REAL subprocess and
hit real HTTP (not mocked): full lifecycle (start -> /health -> status -> stop
-> confirm process actually exited), idempotent start/stop, stale-state-file
recovery, and a smoke pass over the actual TokenOps-facing routes
(runs/governance/events:batch/precheck/halt) through the started instance.
Manually verified the Windows detach/terminate path locally (this repo's CI is
ubuntu-only, so it only exercises the POSIX branch).

Full suite: 42 passed. ruff clean.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…stop)

The repo's first container image. For a shared/hosted plane, not local dev --
kept clearly separate from the start/stop/status work in the previous commit.

- Dockerfile: python:3.12-slim, builds from source (pip install .), non-root
  user, SQLite on a /data volume, built-in HEALTHCHECK against /health.
  Entrypoint is `control-plane serve` (foreground, PID 1) -- deliberately NOT
  `start`: Docker/Kubernetes is already the process supervisor (restart policy,
  health checks, stdout/stderr log collection), and a background/detached mode
  would exit PID 1 as soon as it spawned its child, taking the container down
  with it.
- docker-compose.yml: build + run on :8800 with a named volume for the DB.
- .dockerignore: excludes tests/docs/ui/caches/local db files from the build
  context.
- README: new Docker section explaining the split from start/stop/status and
  why serve is the right entrypoint for a container.

Verified without a Docker daemon (none available here): built the actual wheel
(`python -m build`), confirmed hatchling packages web/ templates+static and
default.yaml correctly, installed the wheel into a clean venv (what the image
does), and ran the full start -> /health -> index page (proves Jinja2 templates
+ static assets resolve) -> stop cycle against it end to end.

Full suite: 42 passed. ruff clean.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@susheem-k
susheem-k merged commit 3a9dbc8 into main Sep 12, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Epic: v0.2.0 — ledger precheck + batch write, fresh runs schema, data_scope (TokenOps remote-only)

1 participant