diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index eb2bea7..bed7fc4 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -96,7 +96,8 @@ Canonical product hierarchy (never reverse): Mutiny (behavioral fuzz-testing engine) → Adapter Layer → OpenAI Agents SDK Adapter (Adapter #1) - → Future adapters (LangGraph, PydanticAI, CrewAI, AutoGen, HTTP, …) + → PydanticAI Adapter (source package; CLI-local) + → Future adapters (LangGraph, CrewAI, AutoGen, HTTP, …) → Customer Project ``` @@ -231,10 +232,12 @@ remains the engine; optional Hosted sync uploads redacted local results. These paths are roadmap placeholders, not shipped adapter locations. Adapter #1 is in `packages/mutiny_openai_agents`; the empty `integrations/` directory is tracked separately in [issue #38](https://github.com/CodewithJha/mutiny/issues/38). +The additional [PydanticAI adapter](../packages/mutiny_pydantic_ai/README.md) +lives in `packages/mutiny_pydantic_ai` and is implemented but not yet published. | Path | Role | |---|---| -| LangGraph / CrewAI / PydanticAI / HTTP adapters | Roadmap (Beta/v1) — see [ROADMAP](./ROADMAP.md) | +| LangGraph / CrewAI / HTTP adapters | Roadmap (Beta/v1) — see [ROADMAP](./ROADMAP.md) | | `integrations/mcp` | MCP tool wrappers around API/Core (roadmap) | | `integrations/skills` | Instructional Skill markdown | @@ -353,8 +356,8 @@ These exist to prevent demo failure and architecture drift. Override only via AD | Constraint | Rule | |---|---| -| Supported adapter | Adapter #1: OpenAI Agents SDK (Core supports more via same interface) | -| LangGraph, CrewAI, PydanticAI, AutoGen, MCP targets | Out of current scope — ROADMAP | +| Supported adapters | OpenAI Agents SDK; PydanticAI source package (unreleased). Both implement the same Core port and execute locally. | +| LangGraph, CrewAI, AutoGen, MCP targets | Out of current scope — ROADMAP | --- diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 7260761..3a61231 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -38,7 +38,8 @@ **Limitations / out of scope (explicit):** -- LangGraph, CrewAI, PydanticAI, AutoGen, HTTP adapters (Beta/v1 — same interface) +- LangGraph, CrewAI, AutoGen, HTTP adapters (Beta/v1 — same interface) +- PydanticAI adapter: implemented in source; see the Beta entry below (not yet published) - MCP (unless stretch after green) - Multi-tenant cloud - Broad repo scanning @@ -67,7 +68,9 @@ New adapters on the **same** `TargetAdapter` interface (Core unchanged): - **LangGraph** adapter - **CrewAI** adapter -- **PydanticAI** adapter +- **PydanticAI** adapter: [implemented in source](../packages/mutiny_pydantic_ai/README.md), + with offline function-tool observation and fail-closed unsupported-tool handling. + CLI-local; Hosted observe-only. PyPI release remains pending. - Localhost OpenAI-compatible **HTTP endpoint adapter** with token gate - Multi-campaign history UX - Policy editor (still deterministic primitives) diff --git a/packages/mutiny_pydantic_ai/README.md b/packages/mutiny_pydantic_ai/README.md new file mode 100644 index 0000000..dd409c9 --- /dev/null +++ b/packages/mutiny_pydantic_ai/README.md @@ -0,0 +1,82 @@ +# Mutiny PydanticAI adapter (unreleased source package) + +`PydanticAIAdapter` implements Mutiny's `TargetAdapter` for **local** PydanticAI +agents. Core is framework-independent. Hosted remains observe-only: customer +agents run on your machine; `mutiny run --hosted` uploads redacted lineage. + +## Install from this checkout + +```bash +uv sync --group dev +``` + +For an existing environment, install the local Core and adapter packages: + +```bash +pip install -e packages/mutiny_core -e packages/mutiny_pydantic_ai +``` + +This package uses `pydantic-ai-slim>=2.61.0,<3` (tested with 2.61.0). Install your +chosen PydanticAI provider extra separately for live models. The offline example +below needs no provider, credentials, or network access. This package is not yet +advertised as available on PyPI. + +## Wire a local project + +Create `.mutiny/adapter.py` in the project containing your PydanticAI `Agent`: + +```python +from agent import support_agent +from mutiny_pydantic_ai import PydanticAIAdapter + +def create_adapter(): + return PydanticAIAdapter( + support_agent, + context={"customer": {"email": "alice@example.com"}}, + ) +``` + +Run `mutiny init` if needed, then edit `policy.yaml` for the actual tools your +agent exposes. Keep the custom adapter factory when scaffolding. Run +`mutiny run`, minimize findings through the existing local campaign flow, and +replay saved findings with `mutiny test`. No CLI target enum or Hosted execution +path is added; the CLI already loads any local `TargetAdapter` factory. + +## Offline tool observation + +```python +from pydantic_ai import Agent +from pydantic_ai.models.test import TestModel +from mutiny_pydantic_ai import PydanticAIAdapter + +agent = Agent(TestModel()) + +@agent.tool_plain +def issue_refund(amount: int) -> dict: + return {"refunded": amount} # Mock only: no real refund. + +adapter = PydanticAIAdapter(agent) +turn = adapter.step("demo", "Please issue a refund") +assert turn.tool_calls[0].name == "issue_refund" +assert turn.tool_results[0]["call_id"] == turn.tool_calls[0].id +``` + +The adapter consumes PydanticAI's public function-tool events. Only validated +calls that execute are mapped to tool evidence; output-schema tools, argument +validation failures, and skipped calls do not become function-tool invocations. +No tool events on a text-only turn is valid. Missing observation surfaces, +unpaired executed calls, or provider-side built-in tool calls raise +`ToolsNotObservableError`. Tool evidence is never inferred from assistant text. + +History is isolated per session. `reset(session_id)` clears that conversation; +use `on_reset` to reset mock tools or other external state. `deps` is passed to +the agent separately from the deterministic policy facts returned by `context`. +`context_provider` can supply fresh policy facts; returned facts are copied. +`request_limit` bounds model requests per turn (default 10). SDK failures +propagate to the existing candidate error handling. The SDK runs synchronously +inside the local runner; share neither an adapter nor stateful tools between +concurrent campaigns. + +Public SDK references: [agent execution](https://pydantic.dev/docs/ai/api/pydantic-ai/agent/), +[tool events](https://pydantic.dev/docs/ai/api/pydantic-ai/messages/), and +[offline testing](https://pydantic.dev/docs/ai/guides/testing/). diff --git a/packages/mutiny_pydantic_ai/pyproject.toml b/packages/mutiny_pydantic_ai/pyproject.toml new file mode 100644 index 0000000..b34a823 --- /dev/null +++ b/packages/mutiny_pydantic_ai/pyproject.toml @@ -0,0 +1,21 @@ +[project] +name = "mutiny-pydantic-ai" +version = "0.2.0" +description = "Mutiny TargetAdapter for local PydanticAI agents" +readme = "README.md" +requires-python = ">=3.11" +license = "MIT" +dependencies = [ + "mutiny-core>=0.2.0", + "pydantic-ai-slim>=2.61.0,<3", +] + +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[tool.hatch.build.targets.wheel] +packages = ["src/mutiny_pydantic_ai"] + +[tool.hatch.build.targets.sdist] +include = ["/src", "/README.md", "/pyproject.toml"] diff --git a/packages/mutiny_pydantic_ai/src/mutiny_pydantic_ai/__init__.py b/packages/mutiny_pydantic_ai/src/mutiny_pydantic_ai/__init__.py new file mode 100644 index 0000000..79e528e --- /dev/null +++ b/packages/mutiny_pydantic_ai/src/mutiny_pydantic_ai/__init__.py @@ -0,0 +1,5 @@ +"""Local PydanticAI adapter for the framework-independent Mutiny port.""" + +from mutiny_pydantic_ai.adapter import PydanticAIAdapter + +__all__ = ["PydanticAIAdapter"] diff --git a/packages/mutiny_pydantic_ai/src/mutiny_pydantic_ai/adapter.py b/packages/mutiny_pydantic_ai/src/mutiny_pydantic_ai/adapter.py new file mode 100644 index 0000000..c83d32a --- /dev/null +++ b/packages/mutiny_pydantic_ai/src/mutiny_pydantic_ai/adapter.py @@ -0,0 +1,149 @@ +"""Observe local PydanticAI function-tool executions through SDK events.""" + +from __future__ import annotations + +import copy +from collections.abc import Callable +from typing import Any + +from pydantic_ai import Agent, DeferredToolRequests +from pydantic_ai.messages import ( + NativeToolCallPart, + FunctionToolCallEvent, + FunctionToolResultEvent, + ModelMessage, + OutputToolCallEvent, + ToolCallPart, +) +from pydantic_ai.usage import UsageLimits +from pydantic_core import to_jsonable_python + +from mutiny_core.adapter import TargetAdapter, ToolsNotObservableError +from mutiny_core.trace import AdapterTurnResult, ToolCall + + +class PydanticAIAdapter(TargetAdapter): + """Run an Agent locally, retaining separate message history per session. + + Only validated function-tool calls are evidence. Output tools, rejected + arguments, and skipped calls are not executed function tools. Provider-side + built-in tools are unsupported and fail closed rather than hiding evidence. + """ + + def __init__( + self, + agent: Agent[Any, Any], + *, + deps: Any = None, + context: dict[str, Any] | None = None, + context_provider: Callable[[], dict[str, Any]] | None = None, + on_reset: Callable[[], None] | None = None, + request_limit: int = 10, + ) -> None: + if not isinstance(agent, Agent): + raise TypeError("PydanticAIAdapter requires a PydanticAI Agent") + if request_limit < 1: + raise ValueError("request_limit must be positive") + self._agent = agent + self._deps = deps + self._context = copy.deepcopy(context or {}) + self._context_provider = context_provider + self._on_reset = on_reset + self._request_limit = request_limit + self._sessions: dict[str, list[ModelMessage]] = {} + + def reset(self, session_id: str) -> None: + if self._on_reset is not None: + self._on_reset() + self._sessions[session_id] = [] + + def step(self, session_id: str, user_message: str) -> AdapterTurnResult: + if session_id not in self._sessions: + self.reset(session_id) + calls: list[ToolCall] = [] + outputs: list[dict[str, Any]] = [] + observed_call_ids: set[str] = set() + observed = False + + async def observe(_context: Any, events: Any) -> None: + nonlocal observed + observed = True + async for event in events: + if isinstance(event, (FunctionToolCallEvent, OutputToolCallEvent)): + observed_call_ids.add(event.tool_call_id) + if ( + isinstance(event, FunctionToolCallEvent) + and event.args_valid is True + ): + part = event.part + if not part.tool_call_id or not part.tool_name: + raise ToolsNotObservableError( + "PydanticAI tool identity is missing" + ) + calls.append( + ToolCall( + id=part.tool_call_id, + name=part.tool_name, + arguments=part.args_as_dict(), + ) + ) + elif isinstance(event, FunctionToolResultEvent): + # Skipped/invalid calls have results too; pair only actual calls. + if any(call.id == event.tool_call_id for call in calls): + outputs.append( + { + "call_id": event.tool_call_id, + "output": to_jsonable_python(event.part.content), + } + ) + + result = self._agent.run_sync( + user_message, + message_history=copy.deepcopy(self._sessions[session_id]), + deps=self._deps, + usage_limits=UsageLimits(request_limit=self._request_limit), + event_stream_handler=observe, + ) + if not observed or not callable(getattr(result, "new_messages", None)): + raise ToolsNotObservableError( + "PydanticAI run did not expose tool observation" + ) + messages = result.new_messages() + if isinstance(result.output, DeferredToolRequests): + raise ToolsNotObservableError( + "PydanticAI deferred tools require external execution" + ) + if any( + isinstance(part, NativeToolCallPart) + for msg in messages + for part in msg.parts + ): + raise ToolsNotObservableError( + "PydanticAI provider-side built-in tools are unsupported" + ) + if any( + isinstance(part, ToolCallPart) + and part.tool_call_id not in observed_call_ids + for msg in messages + for part in msg.parts + ): + raise ToolsNotObservableError( + "PydanticAI tool call is missing its observation event" + ) + if any(not any(out["call_id"] == call.id for out in outputs) for call in calls): + raise ToolsNotObservableError( + "PydanticAI tool call has no observable result" + ) + turn = AdapterTurnResult( + assistant_message=str(result.output) if result.output is not None else None, + tool_calls=calls, + tool_results=outputs, + raw={"adapter": "pydantic_ai", "session_id": session_id}, + ) + self._sessions[session_id].extend(copy.deepcopy(messages)) + return turn + + def context(self, session_id: str | None = None) -> dict[str, Any]: + _ = session_id + facts = self._context_provider() if self._context_provider else self._context + return copy.deepcopy(facts) diff --git a/pyproject.toml b/pyproject.toml index 2000b45..f36d854 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -3,6 +3,7 @@ dev = [ "mutiny-core", "mutiny-openai-agents", + "mutiny-pydantic-ai", "mutiny-ai", "demo-agent", "mutiny-api", @@ -17,6 +18,7 @@ llm = [ [tool.uv.sources] mutiny-core = { workspace = true } mutiny-openai-agents = { workspace = true } +mutiny-pydantic-ai = { workspace = true } mutiny-ai = { workspace = true } demo-agent = { workspace = true } mutiny-api = { workspace = true } @@ -25,6 +27,7 @@ mutiny-api = { workspace = true } members = [ "packages/mutiny_core", "packages/mutiny_openai_agents", + "packages/mutiny_pydantic_ai", "packages/mutiny_cli", "apps/demo_agent", "apps/api", @@ -43,6 +46,7 @@ testpaths = ["tests"] pythonpath = [ "packages/mutiny_core/src", "packages/mutiny_openai_agents/src", + "packages/mutiny_pydantic_ai/src", "packages/mutiny_cli/src", "apps/demo_agent/src", "apps/api/src", diff --git a/tests/unit/test_package_release.py b/tests/unit/test_package_release.py index ece96fb..555c152 100644 --- a/tests/unit/test_package_release.py +++ b/tests/unit/test_package_release.py @@ -13,6 +13,7 @@ PACKAGES = { "mutiny-core": REPO / "packages" / "mutiny_core", "mutiny-openai-agents": REPO / "packages" / "mutiny_openai_agents", + "mutiny-pydantic-ai": REPO / "packages" / "mutiny_pydantic_ai", "mutiny-ai": REPO / "packages" / "mutiny_cli", } _VERSION_RE = re.compile(r'^version\s*=\s*"([^"]+)"', re.M) @@ -71,6 +72,7 @@ def test_built_wheels_contain_current_surface(tmp_path: Path) -> None: for name, pkg in ( ("core", PACKAGES["mutiny-core"]), ("openai", PACKAGES["mutiny-openai-agents"]), + ("pydantic-ai", PACKAGES["mutiny-pydantic-ai"]), ("cli", PACKAGES["mutiny-ai"]), ): target = out / name @@ -84,6 +86,16 @@ def test_built_wheels_contain_current_surface(tmp_path: Path) -> None: core_whl = next((out / "core").glob("mutiny_core-*-py3-none-any.whl")) openai_whl = next((out / "openai").glob("mutiny_openai_agents-*-py3-none-any.whl")) cli_whl = next((out / "cli").glob("mutiny_ai-*-py3-none-any.whl")) + pydantic_whl = next((out / "pydantic-ai").glob("mutiny_pydantic_ai-*-py3-none-any.whl")) + + with zipfile.ZipFile(pydantic_whl) as z: + names = set(z.namelist()) + assert "mutiny_pydantic_ai/adapter.py" in names + metadata_path = next(n for n in names if n.endswith(".dist-info/METADATA")) + metadata = z.read(metadata_path).decode("utf-8") + assert "Requires-Dist: mutiny-core>=0.2.0" in metadata + assert "Requires-Dist: pydantic-ai-slim" in metadata + assert not any(n.startswith("mutiny_api/") for n in names) with zipfile.ZipFile(core_whl) as z: names = set(z.namelist()) diff --git a/tests/unit/test_pydantic_ai_adapter.py b/tests/unit/test_pydantic_ai_adapter.py new file mode 100644 index 0000000..8fc130b --- /dev/null +++ b/tests/unit/test_pydantic_ai_adapter.py @@ -0,0 +1,318 @@ +"""Offline PydanticAI evidence and session contracts; no provider credentials.""" + +import asyncio +from types import SimpleNamespace + +import pytest +from pydantic import BaseModel +from pydantic_ai import Agent, models +from pydantic_ai.models.test import TestModel +from pydantic_ai.exceptions import UsageLimitExceeded +from pydantic_ai.messages import ( + FunctionToolCallEvent, + FunctionToolResultEvent, + ModelResponse, + NativeToolCallPart, + ToolCallPart, + ToolReturnPart, +) + +from mutiny_core import ToolsNotObservableError +from mutiny_core.adapter import execute_conversation +from mutiny_pydantic_ai import PydanticAIAdapter + + +@pytest.fixture(autouse=True) +def no_live_models(monkeypatch): + monkeypatch.setattr(models, "ALLOW_MODEL_REQUESTS", False) + + +def make_agent(): + calls = [] + agent = Agent(TestModel()) + + @agent.tool_plain + def issue_refund(amount: int, approved: bool) -> dict: + calls.append({"amount": amount, "approved": approved}) + return {"refunded": amount, "approved": approved} + + return agent, calls + + +def test_real_tool_execution_matches_observed_arguments_and_result(): + agent, executed = make_agent() + adapter = PydanticAIAdapter(agent) + turn = adapter.step("s", "Refund please") + assert len(executed) == len(turn.tool_calls) == 1 + call = turn.tool_calls[0] + assert call.name == "issue_refund" + assert call.arguments == executed[0] + assert turn.tool_results == [ + { + "call_id": call.id, + "output": { + "refunded": executed[0]["amount"], + "approved": executed[0]["approved"], + }, + } + ] + assert turn.assistant_message + + +def test_history_is_per_session_and_reset_clears_it(monkeypatch): + agent, executed = make_agent() + histories = [] + resets = [] + original = Agent.run_sync + + def record(self, *args, **kwargs): + histories.append(list(kwargs["message_history"])) + return original(self, *args, **kwargs) + + monkeypatch.setattr(Agent, "run_sync", record) + adapter = PydanticAIAdapter(agent, on_reset=lambda: resets.append(True)) + first = adapter.step("a", "First") + before = len(executed) + second = adapter.step("a", "Second") + assert len(second.tool_calls) == len(executed) - before + adapter.step("b", "Other conversation") + adapter.reset("a") + adapter.step("a", "Fresh conversation") + assert histories[0] == histories[2] == histories[3] == [] + assert histories[1] + assert len(resets) == 3 + assert first.tool_calls + + +def test_text_only_turn_has_no_invented_tool_evidence(): + adapter = PydanticAIAdapter(Agent(TestModel(custom_output_text="Hello"))) + turn = adapter.step("text", "Hello") + assert turn.assistant_message == "Hello" + assert turn.tool_calls == turn.tool_results == [] + + +def test_structured_output_tool_is_not_a_function_tool(): + class Answer(BaseModel): + answer: int + + adapter = PydanticAIAdapter(Agent(TestModel(), output_type=Answer)) + turn = adapter.step("structured", "Answer please") + assert turn.assistant_message + assert turn.tool_calls == turn.tool_results == [] + + +def test_unobservable_run_fails_closed(monkeypatch): + agent, _ = make_agent() + monkeypatch.setattr( + Agent, + "run_sync", + lambda *a, **k: SimpleNamespace( + output="I refunded you", + new_messages=lambda: [], + ), + ) + with pytest.raises(ToolsNotObservableError, match="observation"): + PydanticAIAdapter(agent).step("blind", "Refund") + + +def test_context_is_copied_and_separate_from_agent_deps(): + facts = {"customer": {"email": "alice@example.com"}} + adapter = PydanticAIAdapter(Agent(TestModel()), context_provider=lambda: facts) + returned = adapter.context("s") + returned["customer"]["email"] = "changed" + assert adapter.context()["customer"]["email"] == "alice@example.com" + facts["customer"]["email"] = "new@example.com" + assert adapter.context()["customer"]["email"] == "new@example.com" + + +def test_agent_dependencies_are_passed_to_tools(): + from pydantic_ai import RunContext + + agent = Agent(TestModel(), deps_type=dict) + + @agent.tool + def customer_name(ctx: RunContext[dict]) -> str: + return ctx.deps["name"] + + adapter = PydanticAIAdapter(agent, deps={"name": "Alice"}, context={"limit": 200}) + assert adapter.step("s", "Who am I?").tool_results[0]["output"] == "Alice" + assert adapter.context() == {"limit": 200} + + +def test_existing_local_cli_loader_accepts_factory(tmp_path): + from mutiny_openai_agents.loader import load_adapter_factory + + (tmp_path / ".mutiny").mkdir() + (tmp_path / ".mutiny" / "adapter.py").write_text( + "from pydantic_ai import Agent\n" + "from pydantic_ai.models.test import TestModel\n" + "from mutiny_pydantic_ai import PydanticAIAdapter\n" + "def create_adapter():\n" + " return PydanticAIAdapter(Agent(TestModel(custom_output_text='local')) )\n", + encoding="utf-8", + ) + adapter = load_adapter_factory(tmp_path)() + assert isinstance(adapter, PydanticAIAdapter) + assert adapter.step("cli", "Hi").assistant_message == "local" + + +def test_existing_runner_can_use_adapter_in_worker_thread(): + agent, executed = make_agent() + trace = execute_conversation( + PydanticAIAdapter(agent), + ["Refund", "Thanks"], + candidate_id="c", + session_id="s", + ) + assert trace.status == "scored", trace.error + assert len(trace.all_tool_calls) == len(executed) + assert trace.all_tool_calls + + +def test_request_limit_is_enforced(): + agent, _ = make_agent() + with pytest.raises(UsageLimitExceeded, match="request_limit"): + PydanticAIAdapter(agent, request_limit=1).step("limit", "Refund") + + +@pytest.mark.parametrize("limit", [0, -1]) +def test_request_limit_must_be_positive(limit): + with pytest.raises(ValueError, match="positive"): + PydanticAIAdapter(Agent(TestModel()), request_limit=limit) + + +def run_with_events(monkeypatch, events, messages=(), output="done"): + async def stream(): + for event in events: + yield event + + def run(self, *args, **kwargs): + asyncio.run(kwargs["event_stream_handler"](None, stream())) + return SimpleNamespace(output=output, new_messages=lambda: list(messages)) + + monkeypatch.setattr(Agent, "run_sync", run) + return PydanticAIAdapter(Agent(TestModel())) + + +@pytest.mark.parametrize("args_valid", [False, None]) +def test_invalid_and_skipped_tool_calls_are_not_execution_evidence( + monkeypatch, args_valid +): + call = ToolCallPart("issue_refund", {"amount": 42}, tool_call_id="not-executed") + adapter = run_with_events( + monkeypatch, + [ + FunctionToolCallEvent(call, args_valid=args_valid), + FunctionToolResultEvent( + ToolReturnPart("issue_refund", "skipped", tool_call_id="not-executed") + ), + ], + ) + turn = adapter.step("s", "Refund") + assert turn.tool_calls == turn.tool_results == [] + + +def test_executed_tool_without_result_fails_closed(monkeypatch): + adapter = run_with_events( + monkeypatch, + [ + FunctionToolCallEvent( + ToolCallPart( + "issue_refund", {"amount": 42}, tool_call_id="missing-return" + ), + args_valid=True, + ) + ], + ) + with pytest.raises(ToolsNotObservableError, match="result"): + adapter.step("s", "Refund") + + +def test_provider_side_tool_fails_closed(monkeypatch): + adapter = run_with_events( + monkeypatch, + [], + messages=[ + ModelResponse( + parts=[ + NativeToolCallPart("web_search", {}, tool_call_id="native"), + ] + ) + ], + ) + with pytest.raises(ToolsNotObservableError, match="provider-side"): + adapter.step("s", "Search") + + +def test_transcript_tool_with_missing_event_fails_closed(monkeypatch): + adapter = run_with_events( + monkeypatch, + [], + messages=[ + ModelResponse( + parts=[ + ToolCallPart("issue_refund", {"amount": 42}, tool_call_id="hidden"), + ] + ) + ], + ) + with pytest.raises(ToolsNotObservableError, match="missing its observation"): + adapter.step("s", "Refund") + + +def test_deferred_tool_execution_fails_closed(monkeypatch): + from pydantic_ai import DeferredToolRequests + + adapter = run_with_events(monkeypatch, [], output=DeferredToolRequests()) + with pytest.raises(ToolsNotObservableError, match="deferred"): + adapter.step("s", "Refund") + + +def test_campaign_minimize_and_regression_work_without_core_changes(): + from mutiny_core import CampaignConfig, CampaignEngine, PolicySet + from mutiny_core.minimize import minimize_genome + from mutiny_core.regress import replay_regression, save_regression + + agent, _ = make_agent() + adapter = PydanticAIAdapter(agent) + policy = PolicySet.model_validate( + { + "version": "1", + "target": "pydantic_ai", + "rules": [ + { + "id": "no_refund", + "tool": "issue_refund", + "kind": "deny_tool", + "description": "No refunds", + } + ], + } + ) + result = CampaignEngine( + adapter=adapter, + policy_set=policy, + config=CampaignConfig(population_size=2, elite_count=1, max_generations=1), + ).run() + assert result.violated + assert result.best is not None + minimized = minimize_genome( + result.best.genome, + adapter=adapter, + policy_set=policy, + target_rule_ids=["no_refund"], + campaign_id="offline", + candidate_id=result.best.genome.id, + ) + assert minimized.still_reproduces + regression = save_regression( + minimized, name="refund", target="pydantic_ai", policy_set=policy + ) + assert ( + replay_regression(regression, adapter=adapter, policy_set=policy).status + == "FAIL" + ) + safe = PydanticAIAdapter(Agent(TestModel(custom_output_text="Refund refused"))) + assert ( + replay_regression(regression, adapter=safe, policy_set=policy).status == "PASS" + ) diff --git a/uv.lock b/uv.lock index e1f9232..00c89e1 100644 --- a/uv.lock +++ b/uv.lock @@ -16,6 +16,7 @@ members = [ "mutiny-api", "mutiny-core", "mutiny-openai-agents", + "mutiny-pydantic-ai", ] [manifest.dependency-groups] @@ -25,6 +26,7 @@ dev = [ { name = "mutiny-api", editable = "apps/api" }, { name = "mutiny-core", editable = "packages/mutiny_core" }, { name = "mutiny-openai-agents", editable = "packages/mutiny_openai_agents" }, + { name = "mutiny-pydantic-ai", editable = "packages/mutiny_pydantic_ai" }, { name = "pytest", specifier = ">=8.0" }, { name = "pytest-asyncio", specifier = ">=0.24" }, { name = "ruff", specifier = "==0.17.0" }, @@ -361,6 +363,19 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/bd/f4/27e386913417ad32aae42bba48b0c0cce40e9ff2fba1a871ca2702c37324/fastapi-0.143.0-py3-none-any.whl", hash = "sha256:3e9395fd35276425b61b516a31fdd7c77fe2af83e41b4da22e30696fb1304c5d", size = 144665, upload-time = "2026-10-08T12:29:44.853Z" }, ] +[[package]] +name = "genai-prices" +version = "0.1.13" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "httpx2" }, + { name = "pydantic" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/ad/de/119c7ba00b9ddbf629de34cf390ab472da6d7bc5df0c51cdbec900280a74/genai_prices-0.1.13.tar.gz", hash = "sha256:76c9e2dd3917fdacdf3bc562297c4591384142ea2283c477e1512fafa4e3a384", size = 134622, upload-time = "2026-10-15T09:47:04.739Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/41/5a/b6b7d55148d367d07bcd3efdf24e3086ad199ea5109d0c335a872dfaac95/genai_prices-0.1.13-py3-none-any.whl", hash = "sha256:6d2b40760a2f7f5e207bac9b98de0af80e60c58ead497872a5331a7cee610477", size = 142541, upload-time = "2026-10-15T09:47:03.424Z" }, +] + [[package]] name = "griffelib" version = "2.1.0" @@ -629,6 +644,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/41/45/1a4ed80516f02155c51f51e8cedb3c1902296743db0bbc66608a0db2814f/jsonschema_specifications-2025.9.1-py3-none-any.whl", hash = "sha256:98802fee3a11ee76ecaca44429fda8a41bff98b00a0f2838151b113f210cc6fe", size = 18437, upload-time = "2025-09-08T01:34:57.871Z" }, ] +[[package]] +name = "logfire-api" +version = "5.3.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/9d/eb/f79e2c30fb46040fe54e9e994d3e93534ccb01756858e98165329ff17652/logfire_api-5.3.0.tar.gz", hash = "sha256:e3e4265c926dd505d7986d1970f5a4d8fb1b20fe033d2ff12b419ca7f163b533", size = 99890, upload-time = "2026-10-23T15:11:02.864Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/da/13/d6c1400252d4c060e5202967a6c123b80537f5694afad8921df7f0fd6b9e/logfire_api-5.3.0-py3-none-any.whl", hash = "sha256:eaa86578338c8feab4064f8bf206391554d0dc95679ca8eee27fccf722541132", size = 151954, upload-time = "2026-10-23T15:10:58.323Z" }, +] + [[package]] name = "mcp" version = "1.29.0" @@ -737,6 +761,21 @@ requires-dist = [ { name = "openai-agents", specifier = ">=0.19" }, ] +[[package]] +name = "mutiny-pydantic-ai" +version = "0.2.0" +source = { editable = "packages/mutiny_pydantic_ai" } +dependencies = [ + { name = "mutiny-core" }, + { name = "pydantic-ai-slim" }, +] + +[package.metadata] +requires-dist = [ + { name = "mutiny-core", editable = "packages/mutiny_core" }, + { name = "pydantic-ai-slim", specifier = ">=2.61.0,<3" }, +] + [[package]] name = "openai" version = "3.26.1" @@ -831,6 +870,25 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/eb/47/c95ffc2009878c7aac0c5e08528022dcb885933252a88b5f170058014464/pydantic-2.13.5-py3-none-any.whl", hash = "sha256:346a034f080da3755d8e9cb5e00e8b07de1d39e4f6e2c87d8ab7cafa0b269a73", size = 472589, upload-time = "2026-08-28T14:03:59.136Z" }, ] +[[package]] +name = "pydantic-ai-slim" +version = "2.61.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "anyio" }, + { name = "genai-prices" }, + { name = "griffelib" }, + { name = "httpx2" }, + { name = "opentelemetry-api" }, + { name = "pydantic" }, + { name = "pydantic-graph" }, + { name = "typing-inspection" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/cc/c9/a50232708b51eea19100fa515ebf2f4bcf92d1110c0cba0abd0aa1117ec6/pydantic_ai_slim-2.61.0.tar.gz", hash = "sha256:fe134b7fb8c65ca1e102bcd4c46bd8253866423f9a639ba7ed2341b6eb34b8f2", size = 2172199, upload-time = "2026-10-24T03:55:35.493Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/05/03/c8795de4456582b9d04f35c0778c8cb0f2e8b432e250c42c86d1844c039b/pydantic_ai_slim-2.61.0-py3-none-any.whl", hash = "sha256:1d1a2b651de2555c6d0e22e14716969db7cc4f7ff5bd0e06578d5fa6648be462", size = 2514560, upload-time = "2026-10-24T03:55:24.447Z" }, +] + [[package]] name = "pydantic-core" version = "2.46.5" @@ -933,6 +991,21 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/db/1d/068464f23075f66a8f1b806935e9cd9363ee446636ea70d2c22ee8659dbf/pydantic_core-2.46.5-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:d0a24b40877af2de4950252be9d21eaf7fb07660f3c2cae1f56c6b599ada5266", size = 2140686, upload-time = "2026-08-28T10:01:28.947Z" }, ] +[[package]] +name = "pydantic-graph" +version = "2.61.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "anyio" }, + { name = "logfire-api" }, + { name = "pydantic" }, + { name = "typing-inspection" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/f8/45/c48137400dffe74ab0a414adb3596493890a376e96d96b5508aad80ff929/pydantic_graph-2.61.0.tar.gz", hash = "sha256:30e93a073998d8bc90ef2b765a8e9b0dc443cca47eb54da78ed8a6986884752d", size = 46274, upload-time = "2026-10-24T03:55:39.161Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/dc/91/6adc6c0e8275343dec885176a24f6bb12164c3ef79c025fe6e3af3895e95/pydantic_graph-2.61.0-py3-none-any.whl", hash = "sha256:e304dc150ffa148ce04c3b9b248d0333c639abb6812c4cf4f5bcaa9a8eec17fc", size = 53439, upload-time = "2026-10-24T03:55:29.873Z" }, +] + [[package]] name = "pydantic-settings" version = "2.14.2"