Skip to content

Ledger.runs (per-run RunState) is never evicted — unbounded memory growth for long-lived / shared-governor processes #115

Description

@susheem-k

Problem

Ledger.open_run(run_id) is the only thing that adds to Ledger.runs: dict[str, RunState] (src/tokenops/control/ledger.py:206, called from run.py:249). There is no close_run, no .pop, no .clear, no TTL sweep anywhere in control/.

Each RunState holds window: list[BoundaryStep], and every BoundaryStep carries input / output snippets (~500 chars each) plus usage, tags, hashes. So an un-evicted run is not tiny.

When it leaks

  • Default path (per-run governor): tokenops_run builds a fresh Governor + Ledger via build_governance_stack each call; the Ledger is dereferenced when the with block exits, so GC reclaims it. No leak here.
  • Shared / long-lived governor: tokenops_run(governor=...) is a supported path (run.py), and docs/concurrency.md points at a future GovernorContainer / shared-Governor model (GovernorContainer: process-global governors keyed by run_id with TTL cleanup #56). With one Ledger reused across many runs, Ledger.runs grows without bound — every run handled by the process stays resident forever.
  • mark_halted / clear_halt also do self.runs[run_id] = RunState() when the entry is missing, so even halt bookkeeping for an unknown run adds a permanent entry.

clear_run_context() on tokenops_run exit only resets the context vars (bound registration / span) — it does not touch Ledger.runs.

Desired behavior

  • Ledger.close_run(run_id) that drops the per-process RunState (window, step count).
  • tokenops_run calls it in its finally for the run it open_run'd (unless the caller opted to keep it, e.g. resumable runs).
  • For shared-governor / container use: a TTL / max-entries sweep on Ledger.runs as a backstop.
  • Cross-check the two-tier RunState design in scratch/remote-only-control-plane-plan.md Part 6 — the local tier is a cache and needs the same eviction.

Acceptance sketch

  • A process that runs N sequential tokenops_run scopes with a shared governor holds O(1) RunState entries, not O(N).
  • close_run is idempotent and safe to call for a run that was never opened.

Related: #56 (GovernorContainer TTL cleanup), plan Part 6.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions