Skip to content

How should cost from agents not instrumented with TokenOps be attributed? #141

Description

@tishachawla-jg

Summary (RFC — scope first, then spec)

TokenOps attributes cost per agent for calls that go through the SDK (wrap_complete / make_on_step). It cannot see spend from agents it doesn't wrap. "Map other agents' cost" could mean three quite different things, each with a different design. This issue exists to pick the real one before writing a spec.

How attribution works today

  • Every governed crossing carries an Attribution (run_id, agent, parent_run, tags) — src/tokenops/control/core.py.
  • The Governor.observe(...) path prices the crossing and writes it to the ledger under that attribution.
  • A2A: a delegated child run links to its caller via parent_run; delegate crossings can roll a child's total up the hop.

So per-agent cost is solved for agents that run the TokenOps SDK. The gap is everything else.

The three things this could mean

A. Uninstrumented calls in your own agent. A code path that calls the model without the governed wrapper (raw providers.complete). Its spend never reaches the ledger. This is the opt-in/bypass gap tracked in #109; attribution can't fix what it never sees.

B. External / opaque agents over A2A. You call a downstream agent or service that runs its own model calls and isn't wrapped. Today only what crosses your boundary is priced; the child's internal spend is invisible unless it reports up.

C. Priced third-party tools. A tool call that itself costs money (a paid search API, a paid retrieval service). It's a crossing, but its dollar cost isn't a token-priced LLM call.

Options (map to the scenarios above)

  1. External-cost ingest API (covers B and C): record_external_cost(run_id, micros, source, agent=...) so an uninstrumented or external component reports spend into the run's ledger under an attribution. Needs auth/ownership (ties to control-plane #6) so a caller can't charge someone else's run.
  2. A2A cost propagation (covers B): make the delegate rollup explicit and documented — a child run returns its priced total, the parent records it under the child's attribution with parent_run lineage. Partly exists; formalize and test it.
  3. Wrap-completeness (covers A): the real answer to A is not attribution but making enforcement non-bypassable (Fail-closed when a run has no governed dispatch, instead of silently running ungoverned #109) — an uninstrumented call is unattributable by definition.
  4. Document the boundary (baseline): state plainly that only governed crossings are attributed, and un-wrapped/external spend is out of scope until 1-3 land.

Acceptance for THIS issue

  • Agree which scenario(s) are the real need (comment with the concrete deployment).
  • Convert the chosen path into its own implementation issue with a spec.

This is intentionally a question / RFC, not a build ticket — the design diverges a lot depending on the answer.

Related

#109 (non-bypassable enforcement), control-plane #6 (ownership authz for any external ingest), A2A delegate rollup in control/core.py / control/ledger.py.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestquestionFurther information is requested

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions