A local CLI for inference-server, gateway, and SDK maintainers debugging the
gap between 200 OK and a client that still fails. Doctor runs bounded
lifecycle checks for OpenAI Chat, Responses, and Anthropic Messages, then
leaves a redacted report you can reproduce, compare, and share.
Quick Start · What it checks · llama.cpp A/B case · Real SDK case · Example failure report · 简体中文
Linux and macOS can install the exact v0.1.1 release without Go:
curl --proto '=https' --tlsv1.2 -fsSL \
https://raw.githubusercontent.com/whyiug/agentapi-doctor/v0.1.1/install.sh | sh
$HOME/.local/bin/doctor demoThe pinned installer verifies the release archive against checksums.txt
before extraction. If you prefer to inspect it first, download
install.sh, then run sh install.sh. Windows users can download
the verified ZIP from GitHub Releases;
the Installation guide includes checksum steps for every
platform.
The demo needs no API key. It starts a random loopback fixture, runs four lifecycle checks, stores local evidence, and stops the fixture automatically:
Result: CHECKS PASSED
Cases: 4 candidate / 4 applicable / 4 executed
Verdicts: PASS 4 | FAIL 0 | WARN 0 | INCONCLUSIVE 0 | SKIPPED 0 | ERRORED 0
Important conditions:
[candidate_interpretations_pending_review] Candidate raw-wire interpretations; not certification.
Demo success validates this exact synthetic fixture and the installed CLI. It does not certify another endpoint, SDK, provider, or deployment.
No project initialization or YAML is required:
export DOCTOR_TOKEN='replace-with-a-test-token'
doctor test \
--base-url 'https://your-endpoint.example/v1' \
--protocol openai-chat \
--model 'your-model-id' \
--auth-env DOCTOR_TOKENUse openai-responses or anthropic-messages for those API shapes. Omit
--auth-env for an unauthenticated endpoint. The endpoint can be local, on a
private network, or remote; it only needs to be yours or explicitly authorized
for testing.
Each run sends at most four requests under one 60-second deadline. The token is
read from the named environment variable, not a command argument. Requests stay
on the configured origin, redirects are not followed, and evidence remains in
the local .agentapi/ directory. Treat that directory as private local state
and add .agentapi/ to the tested project's .gitignore.
| A basic smoke test sees | Doctor also checks |
|---|---|
| HTTP status | Required response envelope |
| First SSE chunk | Stream media type and lifecycle |
| Some generated text | Terminal event presence, status, and exactly-once behavior |
| A transient console log | Content-addressed, secret-redacted evidence bound to the exact run |
Today, each Quick Check selects four executable raw HTTP checks for one of:
- OpenAI Chat Completions;
- OpenAI Responses;
- Anthropic Messages.
It can render the same result as terminal output, JSON, JUnit, SARIF, Markdown, or a self-contained offline HTML report. Named baselines and stable exit codes make the result usable in CI.
In llama.cpp PR #21174, the PR author described an earlier Doctor run as useful independent evidence. We then retested the updated head locally against the same bounded checks:
| Target | Existing OpenAI SDK smoke | Selected PR tests | Doctor v0.1.1 |
|---|---|---|---|
master at test time, f955e394 |
2/2 | 1/9 | 3 PASS + 1 FAIL |
PR head, a28a6d324 |
not separately rerun | 9/9 | 4 PASS |
The master failure was a Responses output item with no nonnegative output index. The same Doctor check and the selected upstream sequence/index tests passed on the PR head. See the commit-pinned case for the exact commits, test blob, binary and model hashes, build boundary, and limitations. This comparison supports those checks only; it is not a merge-readiness or full-compatibility claim.
When the checked-in synthetic server omits the Responses terminal event, Doctor rejects the stream even though its media type looks correct:
Result: CHECKS FAILED
Cases: 4 candidate / 4 applicable / 4 executed
Verdicts: PASS 3 | FAIL 1 | WARN 0 | INCONCLUSIVE 0 | SKIPPED 0 | ERRORED 0
Important conditions:
[candidate_interpretations_pending_review] Candidate raw-wire interpretations; not certification.
PASS stream media type
PASS required response envelope
FAIL terminal event exactly once
PASS terminal status
Download the offline failure report and open it locally, or reproduce it with the documented Synthetic Fixture. This example is a real, deterministic wire/lifecycle observation. It is not by itself a real SDK run or automatic root-cause attribution.
A status-only smoke test can accept 200 OK and text/event-stream without
ever asking whether the terminal object is usable. The source tree now includes
one deliberately narrow counterexample using the real, pinned OpenAI Python SDK:
| Observation of the same synthetic stream | Result |
|---|---|
| HTTP/SSE smoke | 200 OK; a response.completed event arrived |
| Raw terminal object | output is null, not the array modeled by the pinned SDK |
| OpenAI Python SDK 2.38.0 | rejects the stream during event iteration |
| Doctor bundle | correlates wire.sse with the sanitized SDK observation and exact dependency lock |
Reproduce it on Linux amd64 with CPython 3.12.12:
doctor reproduce openai-python-responses \
--python .venv/bin/python \
--fixture null-completed-output \
--bundle ./openai-python-null-output.zipThis command uses a random loopback fixture and a synthetic token; it never contacts a provider or reads an API key. See the reproducible case study for the hash-locked install and exact evidence boundary. The case proves one frozen SDK behavior, not compatibility or incompatibility of any vendor endpoint.
| Your goal | Best tool today |
|---|---|
| Check whether one key or endpoint responds | curl or a browser checker |
| Explore models and prompts | A web playground |
| Repeatedly inspect lifecycle behavior and keep diffable evidence | AgentAPI Doctor |
| Reproduce one known Responses/SDK failure | Use Doctor's pinned OpenAI Python case and evidence bundle |
| Prove an arbitrary SDK or Agent compatible | Run that exact client against the authorized endpoint; Doctor does not claim this coverage |
Doctor is not a model-quality benchmark, provider ranking, relay checker, or vendor certification service. The current catalog also contains candidate metadata that is not executable coverage; see Known Limitations.
- Credentials are resolved from an environment or protected file reference and are redacted before persistence.
- Exact endpoint, model, plan, profile, pack, and evidence digests stay bound to the run so two results can be meaningfully compared.
- Structured model content and tool arguments are not necessarily anonymous. Review evidence before sharing it.
- The provider still receives the bounded synthetic prompts and may retain them under its own policy.
- Structural checks request 64 output tokens. The Chat/Responses terminal-status check requests 512 so default-thinking models can reach a natural terminal more often. A provider may reject or ignore either field, so these are not enforced cost ceilings.
Only test systems you are explicitly authorized to assess.
Real reports repeatedly show the same gap: direct requests work, but an SSE terminal, tool-call delta, strict Responses event, proxy, or client state machine breaks later. Examples include Open WebUI #21768, llama.cpp #20607, and Codex #24973.
The public Roadmap keeps the next work focused on reproducible endpoint checks, report reuse, and verified client cases. Registry, public matrix, and hosted-service candidates remain outside the supported v0.1 scope.
Quick Start · Installation · CLI reference · Troubleshooting · Known limitations · llama.cpp A/B case · Real SDK case · Roadmap · All docs
Contributions are welcome. Read CONTRIBUTING.md, propose a real compatibility failure or fixture, and use SECURITY.md for private vulnerability reports. Please do not open a public issue containing a credential or unredacted provider response.
Source and documentation use the Apache License 2.0 unless a file says otherwise. Vendored dependency notices are recorded in THIRD_PARTY_LICENSES.txt.