Repository navigation
feat(adapter): add local PydanticAI TargetAdapter with offline tool evidence (#45) - #139
Conversation
|
@BasilZafar11 is attempting to deploy a commit to the priyanshu's projects Team on Vercel. A member of the Team first needs to authorize it. |
CodewithJha
left a comment
There was a problem hiding this comment.
I reviewed this against #45's acceptance criteria and ran it.
Acceptance criteria
- Package and README, CLI-local with Hosted observe-only: solved. The README is clear that this is an unreleased source package.
- Offline test with tool observation, plus an explicit observability error contract: solved, both. The tests use the real SDK (
TestModel) for the happy path and assertToolsNotObservableErrorfor six unobservable paths. - Core unchanged: confirmed.
git diff maintouches nothing underpackages/mutiny_coreorapps/api. The only shared files are the workspacepyproject.toml,uv.lock(additions only), the package-release test and the docs.
What I ran
uv run ruff checkpasses. The full suite withMUTINY_SAMPLE_OFFLINE=1gives 758 passed. The 19 adapter tests and the wheel-metadata check pass.- My own probe with a streamed
FunctionModel: two parallel tool calls come back with their ids, args and results. AModelRetryis recorded as an executed call whose result is the retry text. An uncaught tool exception propagates as the original error. Session history is only advanced after a successful, fully observed turn, so a fail-closed error doesn't corrupt the session.
What I liked
- Evidence is built only from validated
FunctionToolCallEvents (args_valid is True). Output tools, rejected args and skipped calls are not counted as executed tools. A policy engine that trusts this trace can't be fed phantom calls. - It fails closed in the right places: native provider-side tools, deferred external execution, a transcript tool call with no event, and an executed call with no result.
- Tests cover reset and per-session history isolation, deps passed through, context kept separate from deps,
request_limit, loading through the existing CLI factory loader, and the full campaign, minimize and regression FAIL/PASS loop. That last one shows the port really holds without Core changes.
Non-blocking notes (no change needed to merge)
- Passing
event_stream_handlermakes PydanticAI use streaming model requests. Models or test doubles with no streaming support will fail at run time. A sentence in the README would save a user some confusion. - Recording a
ModelRetry'd call as executed is defensible. It is worth a one-line mention in the README so policy authors know what they will see. - The package isn't in
verify_release_artifacts.shorpublish.yml, which matches the "unreleased" framing. Publishing will need that wiring as a separate step.
I'll add the CHANGELOG entry myself after merge.
|
Merged. Thanks for the contribution. You handled the part that matters for an adapter well: tool evidence comes only from validated function-tool events, so rejected, skipped or output-tool calls can't show up as executed tools. Everything the adapter can't observe fails closed. The campaign, minimize and regression test running through the unchanged Core port is good proof that the abstraction holds. One correction to my review: the tests cover five unobservable paths, not six. This closes #45. I'll add the CHANGELOG entry. If you want a small follow-up, the README notes on streaming model requests and on how If you'd like to continue working on Mutiny, these would be good related areas to look at:
No pressure on any of them. |
Summary
Fixes #45
Add the independent
mutiny-pydantic-aisource package implementingTargetAdapterfor local PydanticAI agents. It consumes public SDK function-tool events, preserving tool IDs, arguments, and results, and isolates message history by session. It supports agent dependencies, deterministic policy facts, reset hooks, and a model-request limit.Observability fails closed: missing event streams, transcript calls without events, executed calls without results, native provider-side tools, and deferred external execution raise
ToolsNotObservableError. Output-schema tools, failed argument validation, and skipped calls are excluded from executed function-tool evidence. Text-only runs remain valid.Core and Hosted implementation files are unchanged. Existing local adapter factory loading, campaign evaluation, minimization, and regression replay work through the same port. The README documents local project wiring, mock-only offline usage, SDK compatibility, and Hosted observe-only behavior. This is an unreleased source package, not a claim of PyPI availability.
Register the package in the workspace/dev group, lock
pydantic-ai-slim2.61.0 (supported range >=2.61.0,<3), add wheel-surface checks, and label the implemented source adapter in architecture/roadmap docs. Provider extras remain the caller's choice.Type of change
Checklist
Test plan
MUTINY_SAMPLE_OFFLINE=1 uv run --locked pytest -q --tb=short: 758 passed (complete suite, including unit, integration, reliability, and packaging checks).uv run --locked ruff check: passed.git diff --check: passed.No live-provider compatibility is claimed beyond the tested SDK APIs and offline model fixture. Standalone web build, CLI shell smoke, and isolated package artifact shell checks are left to existing CI. Branch starts at upstream main
5ae4b0e, matching fork main at the final fetch.