diff --git a/README.md b/README.md index 8e50429..4153ea4 100644 --- a/README.md +++ b/README.md @@ -482,7 +482,7 @@ Re-derived on 2026-07-28 by running each item, not by reading the commit log. - **Cost is recorded when a backend reports one, estimated when it does not.** Both gateways publish the provider's `cost_usd` through the same `llm_output` envelope, the runtime's usage callback writes it onto the node's `end` event, and an agent's `model` events carry the per-call breakdown. A backend that reports no price still falls back to a `RateCard` estimate, and the two figures stay apart — `recorded_cost_usd` is never a guess. Still missing: no tenant on a trace event, so per-tenant attribution is not offered. - **A node's tokens are its own, not the run's movement while it ran.** Worth stating because it was the other way round: an `end` event carried the difference between two readings of the run's *shared* meter, so under fan-out the workers' windows overlapped and each was credited with its siblings' concurrent spend. Three workers costing 8 tokens each traced as 24/16/8, and `metrics` and `cost` agreed on 48 for 24 tokens of real work — doubling the estimated bill purely because the work ran in parallel. Attribution now comes from a per-node scope on the meter, so the same work costs the same serially and in parallel; a hand charge the usage callback never saw still lands on the node that made it. - **A planning round is an envelope, not a measurement.** A `round` event used to carry the planner's `tokens` and the round's `duration_ms`, both of which `metrics`, `cost` and `replay` add on top of node totals — and the planner's spend was already reported by its own `plan` event, so it was counted twice, and a round's duration encloses the plan plus every node it ran. Neither is on the event now; both are on its `state_delta` as `round_tokens` / `round_iterations` / `round_duration_ms`, where no reader sums them. `RoundRecord.iterations` also holds a figure now rather than always `0`. -- **The Claude CLI backend is completion-only.** Tool calling and structured output need one of the OpenAI-wire backends: `openrouter`, `openai`, or a local `ollama`. +- **The Claude CLI backend is completion-only, and an agent node on it is *delegated* rather than governed.** The CLI has no tool-calling wire format, so GraphARC cannot run its own gated loop over it. Rather than refuse, `AgentNode` hands the whole loop to Claude Code's headless agent — which means every tool Claude Code has, under its `bypassPermissions` mode: those calls are not checked by this graph's permission policy, not confined by the sandbox executor, and the token figure is the sub-agent's own rather than one GraphARC metered call by call. The workspace boundary and the wall-clock ceiling still hold. It warns on `DelegatedToolUseWarning` at construction and marks every trace event `executor=delegated`, so a run stays auditable as delegated; filter that warning to an error to get the old refusal back. Structured output still needs an OpenAI-wire backend: `openrouter`, `openai`, or a local `ollama`. - **A session turn is synchronous**, and a runner claim is a claim rather than a lease — nothing reclaims a session whose runner died holding it. - **`.env` is found by walking up parent directories; `grapharc.toml` is not.** The config layer refuses an upward search on purpose — a run must not be governed by a file you did not know about. The credential loader predates that decision and still searches upward, so the thing that *spends money* is discovered more eagerly than the thing that *constrains* it. - **`grapharc run` has no budget unless you give it one.** Set any of `--max-tokens`, `--max-iterations`, `--max-seconds`, or `--max-concurrency`; without them each dimension is unlimited and the gate admits a topology of any worst-case cost. diff --git a/grapharc/cli/delegate.py b/grapharc/cli/delegate.py index 8d70e89..472b459 100644 --- a/grapharc/cli/delegate.py +++ b/grapharc/cli/delegate.py @@ -25,6 +25,7 @@ import shutil import subprocess import uuid +from dataclasses import dataclass from pathlib import Path from grapharc.cli import style @@ -35,6 +36,130 @@ DEFAULT_DELEGATED_TOOLS = ("Read", "Glob", "Grep", "LS", "Edit", "Write", "Bash") + +# --------------------------------------------------------------------------- +# The reusable core. `run_delegated` below is the CLI's presentation of it, and +# `grapharc.harness.agent.AgentNode` calls it directly when its backend is the +# Claude CLI — one implementation, so the two paths cannot drift on what +# actually gets spawned. + + +class DelegationError(Exception): + """A delegated run could not be started, or came back unreadable. + + `reason` is the machine-readable form, written to the trace so a run that + failed here is distinguishable afterwards from one that ran and refused. + """ + + def __init__(self, message: str, *, reason: str) -> None: + super().__init__(message) + self.reason = reason + + +@dataclass(frozen=True) +class DelegatedRun: + """What Claude Code reported back. Every number here is *its* figure. + + `tokens_reported` is named for what it is: the sub-agent's own count, not + something GraphARC metered call by call. Nothing in this object was + observed by the permission engine. + """ + + ok: bool + answer: str + reason: str + turns: int + tokens_reported: int + cost_usd: float | None + session_id: str | None + allowed: list[str] | None # None == no --allowedTools restriction at all + denied: list[str] + + +def delegate_task( + task: str, + *, + workspace: Path, + model: str | None = None, + allow: list[str] | None = None, + deny: list[str] | None = None, + max_turns: int = 20, + max_seconds: float | None = None, + system_prompt: str | None = None, + permission_mode: str | None = None, +) -> DelegatedRun: + """Run one headless `claude -p` agent loop in `workspace`. + + Two separate axes, and conflating them is a trap worth naming. `allow` + controls *which* tools exist; `permission_mode` controls whether the ones + that mutate anything are allowed to run without a human answering a prompt. + Omitting `--allowedTools` does **not** mean "every tool": it means Claude + Code's own default gating, and headless there is nobody to approve a Write, + so the sub-agent reports back that it could not create the file. Measured, + not assumed. `permission_mode="bypassPermissions"` is what actually means + "everything", and it means it literally — no checks at all. + + Either way the caller is responsible for having said so out loud; + `AgentNode` warns at construction and marks every trace event. + """ + binary = shutil.which("claude") + if binary is None: + raise DelegationError( + "the delegated executor shells out to `claude`, which is not on PATH; " + "install Claude Code or use a tool-calling backend", + reason="claude_not_found", + ) + + workspace = Path(workspace).expanduser().resolve() + workspace.mkdir(parents=True, exist_ok=True) + + argv = [binary, "-p", task, "--output-format", "json", "--max-turns", str(max_turns)] + if allow is not None: + argv += ["--allowedTools", ",".join(allow)] + if deny: + argv += ["--disallowedTools", ",".join(deny)] + if permission_mode: + argv += ["--permission-mode", permission_mode] + if model: + argv += ["--model", model] + if system_prompt: + argv += ["--append-system-prompt", system_prompt] + + try: + completed = subprocess.run( + argv, cwd=workspace, capture_output=True, text=True, timeout=max_seconds + ) + except subprocess.TimeoutExpired as exc: + raise DelegationError( + f"max_seconds ({max_seconds}) reached; the delegated run was stopped", + reason="deadline_exceeded", + ) from exc + + try: + report = json.loads(completed.stdout) + except (json.JSONDecodeError, ValueError) as exc: + detail = (completed.stderr or completed.stdout or "").strip()[-500:] + raise DelegationError( + f"claude exited {completed.returncode} without a readable JSON report: {detail}", + reason="unreadable_report", + ) from exc + + usage = report.get("usage") or {} + met = report.get("subtype") == "success" and not report.get("is_error", False) + return DelegatedRun( + ok=met, + answer=str(report.get("result") or "").strip(), + reason="target_met" if met else str(report.get("subtype") or "error"), + turns=int(report.get("num_turns") or 0), + tokens_reported=int(usage.get("input_tokens") or 0) + + int(usage.get("output_tokens") or 0), + cost_usd=report.get("total_cost_usd"), + session_id=report.get("session_id"), + allowed=list(allow) if allow is not None else None, + denied=list(deny or []), + ) + + def run_delegated( task: str, *, @@ -217,4 +342,10 @@ def run_delegated( return EXIT_OK if met else EXIT_FAILED -__all__ = ["DEFAULT_DELEGATED_TOOLS", "run_delegated"] +__all__ = [ + "DEFAULT_DELEGATED_TOOLS", + "DelegatedRun", + "DelegationError", + "delegate_task", + "run_delegated", +] diff --git a/grapharc/harness/agent.py b/grapharc/harness/agent.py index 882f7ea..3afd123 100644 --- a/grapharc/harness/agent.py +++ b/grapharc/harness/agent.py @@ -53,8 +53,10 @@ import types import typing import uuid +import warnings from collections.abc import Callable, Mapping from enum import StrEnum +from pathlib import Path from typing import Any from langchain_core.language_models.chat_models import BaseChatModel @@ -297,6 +299,48 @@ def _coerce_args(raw: Any) -> dict[str, Any]: raise TypeError(f"tool arguments must be a JSON object, got {type(raw).__name__}") + +class DelegatedToolUseWarning(UserWarning): + """An `AgentNode` is running Claude Code's tool loop instead of GraphARC's. + + Its own category so it can be filtered, asserted on in tests, or turned + into an error with `-W error::grapharc.harness.agent.DelegatedToolUseWarning` + by anyone who wants the old refusal back. + """ + + +#: What the delegated loop runs under. `bypassPermissions` is Claude Code's +#: "no checks at all" mode, and it is deliberate: omitting `--allowedTools` +#: leaves its default gating in place, and headless there is no one to approve +#: a Write — the sub-agent simply reports that it could not create the file. +#: "Every tool Claude Code has" only means that with this set. +DELEGATED_PERMISSION_MODE = "bypassPermissions" + +_DELEGATION_WARNING = ( + "agent node {name!r} is backed by the Claude CLI, which has no tool-calling " + "wire format, so GraphARC cannot run its own tool loop over it. The whole " + "loop is delegated to Claude Code's headless agent, which means: it uses " + "EVERY tool Claude Code has (Bash, Write, WebFetch, Task, ...) under its " + "bypassPermissions mode, so those calls are NOT checked by this graph's " + "permission policy, NOT confined by the sandbox executor, and NOT gated by " + "Claude Code's own prompts either. The token figure is what the sub-agent " + "reports rather than what GraphARC metered. The workspace boundary and the wall-clock " + "ceiling still apply. Every trace event from this node is marked " + "executor=delegated so the run stays auditable; use a tool-calling backend " + "(openrouter/*, openai/*, ollama/*) for a governed loop." +) + + +def _is_claude_cli(model: Any) -> bool: + """Is this the Claude CLI backend? + + Matched on `_llm_type` rather than `isinstance`, so this module does not + import the gateway, and rather than "does it lack bind_tools" — which is + also true of `ScriptedChatModel` and would silently delegate every mock. + """ + return getattr(model, "_llm_type", None) == "grapharc-claude-cli" + + class AgentNode: """A tool-using agent loop, shaped as a GraphARC node. @@ -341,6 +385,16 @@ def __init__( self.prompt_fn = prompt_fn self.trace = trace self.max_tool_result_chars = max_tool_result_chars + #: True when the backend is the Claude CLI, which has no tool-calling + #: wire format and therefore cannot be driven as a raw model. The loop + #: is handed to Claude Code instead — see `_run_delegated`. + self.delegated = _is_claude_cli(model) + if self.delegated: + warnings.warn( + _DELEGATION_WARNING.format(name=name), + DelegatedToolUseWarning, + stacklevel=2, + ) @property def writes(self) -> set[str]: @@ -385,6 +439,9 @@ def run(self, prompt: str, ctx: RunContext | None = None) -> AgentResult: run_id=uuid.uuid4().hex[:12], graph=self.name, meter=BudgetMeter(Budget()) ) + if self.delegated: + return self._run_delegated(prompt, ctx) + model = self._bind_tools() messages: list[BaseMessage] = [ SystemMessage(content=self.system_prompt), @@ -524,6 +581,89 @@ def run(self, prompt: str, ctx: RunContext | None = None) -> AgentResult: # -- internals ------------------------------------------------------------ + def _run_delegated(self, prompt: str, ctx: RunContext) -> AgentResult: + """Hand the whole task to Claude Code's headless agent. + + The trade is stated in `_DELEGATION_WARNING` and repeated on every trace + event this writes, because a warning at construction is gone by the time + anyone reads the run back. `executor="delegated"` on the events is what + stops a reader six months later from assuming this graph's permission + policy saw these tool calls. It did not. + + The workspace boundary and the wall-clock ceiling still hold: the CLI is + spawned with `cwd` set to the harness workspace, and `max_seconds` is + enforced from outside by the subprocess timeout. Everything finer than + that is Claude Code's. + """ + from grapharc.cli.delegate import DelegationError, delegate_task + + remaining = ctx.meter.remaining_seconds() if ctx.meter else None + step = 1 + if self.trace is not None: + self.trace.event( + run_id=ctx.run_id, graph=ctx.graph, node=self.name, phase="model", + step=step, thread_id=ctx.thread_id, attempt=ctx.attempt, + state_delta={"executor": "delegated", "tools": "all of Claude Code's", + "permission_mode": DELEGATED_PERMISSION_MODE, + "governed_by": "Claude Code, not this graph's policy"}, + ) + try: + workspace = getattr(self.harness.executor, "workspace", None) + if workspace is None: + raise DelegationError( + "the delegated executor needs a workspace directory, and this " + f"harness's executor ({type(self.harness.executor).__name__}) " + "does not expose one", + reason="no_workspace", + ) + run = delegate_task( + prompt, + workspace=Path(workspace), + max_turns=self.max_iterations, + max_seconds=remaining, + system_prompt=self.system_prompt, + permission_mode=DELEGATED_PERMISSION_MODE, + ) + except DelegationError as exc: + if self.trace is not None: + self.trace.event( + run_id=ctx.run_id, graph=ctx.graph, node=self.name, phase="stop", + step=step, thread_id=ctx.thread_id, attempt=ctx.attempt, + state_delta={"executor": "delegated", "termination_reason": exc.reason}, + error=str(exc), + ) + return AgentResult( + termination_reason=StopReason.ERROR, iterations=0, note=str(exc) + ) + + # The sub-agent's own count, charged so a budget is not simply blind to + # a delegated node — but named `tokens_reported` everywhere it surfaces, + # because GraphARC did not meter these call by call. + if ctx.meter and run.tokens_reported: + ctx.meter.charge_tokens(run.tokens_reported) + reason = StopReason.TARGET_MET if run.ok else StopReason.ERROR + if self.trace is not None: + self.trace.event( + run_id=ctx.run_id, graph=ctx.graph, node=self.name, phase="stop", + step=step, thread_id=ctx.thread_id, attempt=ctx.attempt, + tokens=run.tokens_reported or None, + cost_usd=run.cost_usd, + state_delta={"executor": "delegated", "termination_reason": reason.value, + "turns": run.turns, "tokens_reported": run.tokens_reported, + "session_id": run.session_id}, + ) + return AgentResult( + output=run.answer if run.ok else "", + partial_output="" if run.ok else run.answer, + termination_reason=reason, + iterations=run.turns, + note=( + f"delegated to Claude Code: {run.turns} turn(s), " + f"{run.tokens_reported:,} tokens reported by the sub-agent, " + "tool calls not checked by this graph's policy" + ), + ) + def _bind_tools(self) -> Any: """Bind the policy-filtered tool set. Denied tools are never described.""" schemas = tool_schemas(self.harness) diff --git a/grapharc/stdlib.py b/grapharc/stdlib.py index 414bf7e..dbd1abc 100644 --- a/grapharc/stdlib.py +++ b/grapharc/stdlib.py @@ -23,10 +23,14 @@ not chosen by the model at run time. `apply_change` is the only one that can write, which is what makes it the one worth denying by default. -All but one of the agent kinds need a **tool-calling** backend: `AgentNode` -refuses a model with no `bind_tools` rather than degrading, so the Claude CLI -subscription cannot drive them. `summarize` is the exception — it is toolless by -design, so it binds nothing and runs anywhere. +All but one of the agent kinds want a **tool-calling** backend, because that is +the only way GraphARC can run the loop itself and gate each call. Given the +Claude CLI — which has no tool-calling wire format — `AgentNode` delegates the +whole loop to Claude Code instead, warning at construction and marking the +trace: the fixed allowlists described above do not apply to a delegated run, +because the tools are Claude Code's rather than this registry's. `summarize` is +the exception either way — it is toolless by design, so it binds nothing and +runs anywhere. Registered but denied is the interesting state: **given a model**, `apply_change` is in the registry because changing files is a real capability, and the default diff --git a/tests/test_agent_delegate.py b/tests/test_agent_delegate.py index 0e3be9a..7437721 100644 --- a/tests/test_agent_delegate.py +++ b/tests/test_agent_delegate.py @@ -117,3 +117,136 @@ def test_a_missing_binary_is_exit_2_with_the_reason(tmp_path, monkeypatch, capsy code = _run(tmp_path) assert code == 2 assert "not on PATH" in capsys.readouterr().err + + +# ---- the same delegation, reached from an AgentNode --------------------------- +# +# `AgentNode` used to refuse the Claude CLI outright: it has no tool-calling wire +# format, so GraphARC cannot run its own gated loop over it. It now delegates the +# whole loop to Claude Code instead, which is a genuine widening of the trust +# boundary — so what these gates pin is that the widening is *visible*, at +# construction and afterwards in the trace. + + +def _node(workspace, trace=None, name="worker"): + from grapharc.gateway import get_model + from grapharc.harness import Harness, PermissionPolicy, PermissionRule, ToolRegistry + from grapharc.harness.agent import AgentNode + + harness = Harness( + ToolRegistry(), + PermissionPolicy(rules=[PermissionRule(action="allow", pattern="*")]), + workspace=str(workspace), + ) + with pytest.warns(Warning): + return AgentNode(get_model("claude-cli"), harness, name=name, trace=trace) + + +def test_a_claude_cli_agent_node_warns_loudly_at_construction(tmp_path, fake_claude): + """A silent switch from "refuses" to "runs with every tool and no checks" + is the one thing this must not be. The warning names each thing given up. + """ + from grapharc.gateway import get_model + from grapharc.harness import Harness, PermissionPolicy, PermissionRule, ToolRegistry + from grapharc.harness.agent import AgentNode, DelegatedToolUseWarning + + harness = Harness( + ToolRegistry(), + PermissionPolicy(rules=[PermissionRule(action="allow", pattern="*")]), + workspace=str(tmp_path), + ) + with pytest.warns(DelegatedToolUseWarning) as caught: + node = AgentNode(get_model("claude-cli"), harness, name="worker") + + assert node.delegated is True + text = str(caught[0].message) + for claim in ("EVERY tool", "NOT checked", "NOT confined", "bypassPermissions"): + assert claim in text, f"the warning does not mention {claim!r}: {text}" + + +def test_a_tool_calling_backend_is_not_delegated_and_does_not_warn(tmp_path): + """The mock double must keep running GraphARC's own loop. + + Detection is on `_llm_type`, not on "does this model lack bind_tools" — + `ScriptedChatModel` lacks it too, and matching that way would have silently + delegated every mocked agent in the suite to a real subprocess. + """ + import warnings as _warnings + + from grapharc.gateway import get_model + from grapharc.harness import Harness, PermissionPolicy, PermissionRule, ToolRegistry + from grapharc.harness.agent import AgentNode + + harness = Harness( + ToolRegistry(), + PermissionPolicy(rules=[PermissionRule(action="allow", pattern="*")]), + workspace=str(tmp_path), + ) + with _warnings.catch_warnings(): + _warnings.simplefilter("error") # any warning at all fails this + node = AgentNode(get_model("mock/x", responses=["hi"]), harness, name="m") + assert node.delegated is False + + +def test_the_delegated_node_asks_for_every_tool_and_bypasses_the_prompt( + tmp_path, fake_claude +): + """Two axes, and conflating them was a real bug found by running it. + + Omitting `--allowedTools` does not mean "every tool" — it leaves Claude + Code's own gating on, and headless there is nobody to approve a Write, so + the sub-agent came back reporting it could not create the file. Only + `--permission-mode bypassPermissions` means what "everything Claude Code + has" was chosen to mean. + """ + workspace = tmp_path / "ws" + workspace.mkdir() + _node(workspace).run("do a thing") + + argv = json.loads(fake_claude.read_text()) + assert "--allowedTools" not in argv, "an allowlist would narrow the tool set" + assert "--permission-mode" in argv + assert argv[argv.index("--permission-mode") + 1] == "bypassPermissions" + + +def test_every_delegated_trace_event_says_it_was_delegated(tmp_path, fake_claude): + """The construction warning is gone by the time anyone reads the run back. + + Without this marking, a JSONL reader six months later sees an agent node + that completed and has no way to know its tool calls never reached this + graph's permission policy — which is exactly the claim the project makes + about its traces. + """ + from grapharc.observe.trace import TraceRecorder + + workspace = tmp_path / "ws" + workspace.mkdir() + trace_path = tmp_path / "t.jsonl" + _node(workspace, trace=TraceRecorder(trace_path)).run("do a thing") + + events = [json.loads(line) for line in trace_path.read_text().splitlines() if line.strip()] + assert events, "the delegated run recorded nothing" + for event in events: + delta = event.get("state_delta") or {} + assert delta.get("executor") == "delegated", event + + opening = events[0]["state_delta"] + assert opening["permission_mode"] == "bypassPermissions" + assert "not this graph's policy" in opening["governed_by"] + + +def test_a_delegated_run_charges_the_meter_what_the_sub_agent_reported(tmp_path, fake_claude): + """A budget must not be simply blind to a delegated node — but the figure is + the sub-agent's own, and every name it surfaces under says so. + """ + from grapharc.runtime.budget import Budget, BudgetMeter + from grapharc.runtime.graph import RunContext + + workspace = tmp_path / "ws" + workspace.mkdir() + ctx = RunContext(run_id="r", graph="g", meter=BudgetMeter(Budget())) + result = _node(workspace).run("do a thing", ctx) + + assert ctx.meter.tokens == REPORT["usage"]["input_tokens"] + REPORT["usage"]["output_tokens"] + assert "tokens reported by the sub-agent" in result.note + assert "not checked by this graph's policy" in result.note