Skip to content

feat(adapter): add local PydanticAI TargetAdapter with offline tool evidence (#45) - #139

Merged
CodewithJha merged 1 commit into
CodewithJha:mainfrom
BasilZafar11:codex/issue-45-pydantic-ai-adapter
Oct 27, 2026
Merged

CodewithJha merged 1 commit into
CodewithJha:mainfrom
BasilZafar11:codex/issue-45-pydantic-ai-adapter

Conversation

@BasilZafar11

Copy link
Copy Markdown
Contributor

Summary

Fixes #45

Add the independent mutiny-pydantic-ai source package implementing TargetAdapter for local PydanticAI agents. It consumes public SDK function-tool events, preserving tool IDs, arguments, and results, and isolates message history by session. It supports agent dependencies, deterministic policy facts, reset hooks, and a model-request limit.

Observability fails closed: missing event streams, transcript calls without events, executed calls without results, native provider-side tools, and deferred external execution raise ToolsNotObservableError. Output-schema tools, failed argument validation, and skipped calls are excluded from executed function-tool evidence. Text-only runs remain valid.

Core and Hosted implementation files are unchanged. Existing local adapter factory loading, campaign evaluation, minimization, and regression replay work through the same port. The README documents local project wiring, mock-only offline usage, SDK compatibility, and Hosted observe-only behavior. This is an unreleased source package, not a claim of PyPI availability.

Register the package in the workspace/dev group, lock pydantic-ai-slim 2.61.0 (supported range >=2.61.0,<3), add wheel-surface checks, and label the implemented source adapter in architecture/roadmap docs. Provider extras remain the caller's choice.

Type of change

  • Adapter (new)
  • Tests
  • Docs / developer experience

Checklist

  • Read contributing guidelines, security policy, code of conduct, and support
  • Linked related issue
  • Core stays free of framework SDK imports and is unchanged
  • Meaningful offline tests added
  • Documentation distinguishes source implementation from a PyPI release
  • One adapter concern per PR
  • Authorized testing with mock tools only

Test plan

  • MUTINY_SAMPLE_OFFLINE=1 uv run --locked pytest -q --tb=short: 758 passed (complete suite, including unit, integration, reliability, and packaging checks).
  • Focused adapter + package checks: 24 passed, including 19 adapter tests, a built-wheel metadata check, real SDK TestModel tool observation, structured/text output, session/reset isolation, dependencies, request limits, unsupported observation paths, existing CLI factory loading, and campaign -> minimize -> regression FAIL/PASS.
  • Live PydanticAI model requests are disabled by the test fixture; no provider credentials are needed.
  • uv run --locked ruff check: passed.
  • git diff --check: passed.

No live-provider compatibility is claimed beyond the tested SDK APIs and offline model fixture. Standalone web build, CLI shell smoke, and isolated package artifact shell checks are left to existing CI. Branch starts at upstream main 5ae4b0e, matching fork main at the final fetch.

@vercel

vercel Bot commented Oct 26, 2026

Copy link
Copy Markdown

@BasilZafar11 is attempting to deploy a commit to the priyanshu's projects Team on Vercel.

A member of the Team first needs to authorize it.

@CodewithJha CodewithJha left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed this against #45's acceptance criteria and ran it.

Acceptance criteria

  • Package and README, CLI-local with Hosted observe-only: solved. The README is clear that this is an unreleased source package.
  • Offline test with tool observation, plus an explicit observability error contract: solved, both. The tests use the real SDK (TestModel) for the happy path and assert ToolsNotObservableError for six unobservable paths.
  • Core unchanged: confirmed. git diff main touches nothing under packages/mutiny_core or apps/api. The only shared files are the workspace pyproject.toml, uv.lock (additions only), the package-release test and the docs.

What I ran

  • uv run ruff check passes. The full suite with MUTINY_SAMPLE_OFFLINE=1 gives 758 passed. The 19 adapter tests and the wheel-metadata check pass.
  • My own probe with a streamed FunctionModel: two parallel tool calls come back with their ids, args and results. A ModelRetry is recorded as an executed call whose result is the retry text. An uncaught tool exception propagates as the original error. Session history is only advanced after a successful, fully observed turn, so a fail-closed error doesn't corrupt the session.

What I liked

  • Evidence is built only from validated FunctionToolCallEvents (args_valid is True). Output tools, rejected args and skipped calls are not counted as executed tools. A policy engine that trusts this trace can't be fed phantom calls.
  • It fails closed in the right places: native provider-side tools, deferred external execution, a transcript tool call with no event, and an executed call with no result.
  • Tests cover reset and per-session history isolation, deps passed through, context kept separate from deps, request_limit, loading through the existing CLI factory loader, and the full campaign, minimize and regression FAIL/PASS loop. That last one shows the port really holds without Core changes.

Non-blocking notes (no change needed to merge)

  • Passing event_stream_handler makes PydanticAI use streaming model requests. Models or test doubles with no streaming support will fail at run time. A sentence in the README would save a user some confusion.
  • Recording a ModelRetry'd call as executed is defensible. It is worth a one-line mention in the README so policy authors know what they will see.
  • The package isn't in verify_release_artifacts.sh or publish.yml, which matches the "unreleased" framing. Publishing will need that wiring as a separate step.

I'll add the CHANGELOG entry myself after merge.

@CodewithJha
CodewithJha merged commit 519545b into CodewithJha:main Oct 27, 2026
6 of 7 checks passed

Copy link
Copy Markdown
Owner

Merged. Thanks for the contribution.

You handled the part that matters for an adapter well: tool evidence comes only from validated function-tool events, so rejected, skipped or output-tool calls can't show up as executed tools. Everything the adapter can't observe fails closed. The campaign, minimize and regression test running through the unchanged Core port is good proof that the abstraction holds. One correction to my review: the tests cover five unobservable paths, not six.

This closes #45. I'll add the CHANGELOG entry. If you want a small follow-up, the README notes on streaming model requests and on how ModelRetry calls are recorded would help users.

If you'd like to continue working on Mutiny, these would be good related areas to look at:

No pressure on any of them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

adapters: PydanticAI TargetAdapter package

2 participants