Agentic API Testing

Upload an API definition, tell the app where the API lives, and test it by chatting: "run negative tests for payments", "which endpoints fail most?", "create defects for the failures".

A local LLM (Ollama) understands the request and chooses what to run. Deterministic code does the testing, so results are repeatable and the model can never invent a pass.

Example

Against the bundled sample API (which has two deliberate bugs), using llama3.2:3b on a 4 GB GPU:

You:   Run all tests
Agent: The run_id is 240ea01e-…  This run failed 2 tests out of 9.
       1. POST /payments - the server did not return a 4xx status code when omitting the 'currency' field.
       2. GET /payments/{paymentId} - schema mismatch at 'paymentId': expected a string, received an integer.

You:   Create defects for the failed tests
Agent: Two defects were created for run 240ea01e-…: one for POST /payments (required_fields check),
       one for GET /payments/{paymentId} (schema check).

Without a token, the same run explains why it fails: 4 request(s) were rejected with 401/403, so the auth token may be missing, expired or wrong.

How it works

flowchart LR
    T([Tester]) --> UI[Streamlit UI]
    UI -->|upload spec, set target| API[FastAPI backend]
    UI -->|chat| API
    API --> ING[ingest adapters]
    ING --> DB[(SQLite: catalog, targets, runs, defects)]
    API --> AG[LangGraph ReAct agent + Ollama]
    AG -->|5 tools| TOOLS[mcp_tools]
    TOOLS --> DB
    TOOLS -->|pytest subprocess| LIB[test_library]
    LIB -->|HTTP| TARGET([API under test])

Loading

1. Ingest. An uploaded file is loaded (JSON or YAML), its format is detected from the content, and an adapter converts it to a canonical API model (endpoint, method, required_fields, request_schema, response_schema). Base URL and auth scheme are read from the file to pre-fill the target form. Everything downstream uses only the canonical model.
2. Target. The tester enters the base URL and, if needed, an auth token in the UI, per project. Nothing is configured through environment variables.
3. Agent. A LangGraph ReAct agent picks among five tools based on the request.
4. Test library. run_tests runs the pytest files in test_library/ against the live API in a subprocess. Each file is one category of check, parametrized over the project's endpoints. Results are saved to SQLite.

Design decisions

Decision	Why
The LLM routes and summarizes; code tests	A small local model is unreliable at generating and judging tests. Deterministic checks are repeatable.
project_id is injected by the backend, not requested from the model	Small models forget or invent arguments. Tools never ask the LLM for what the system already knows.
Tokens never reach the LLM	They are entered in the UI, stored by the backend, and passed to the test subprocess through a temp file, not argv or chat.
Adapter-based ingestion	A new input format is one small adapter that emits the canonical model. The agent and tests don't change.
Failure hints computed in code	If requests get 401/403, the tool result says so; the model only relays it.
Negative tests must be conclusive	A "missing field returns 4xx" test is skipped as inconclusive unless the complete request succeeds first, so a 401 can't count as a pass.
Categories are files	Drop test_<name>.py into test_library/ and the agent tool discovers it.

Test categories

Category	Checks
status_codes	A valid request returns 2xx
schema	A successful response body matches the response schema
required_fields	Omitting each required field returns 4xx (negative tests)

Quick start

Requirements: Python 3.12, Ollama with a tool-calling model.

ollama pull llama3.2:3b

python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -r requirements.txt

Run three processes (three terminals):

# 1. Sample API under test. It requires the bearer token "secret" and has two seeded bugs.
$env:SAMPLE_TARGET_TOKEN = "secret"
python -m uvicorn examples.sample_target:app --port 8080

# 2. Backend
$env:APP_OLLAMA_BASE_URL = "http://localhost:11434"
python -m uvicorn backend.main:app --port 8000

# 3. Frontend
$env:BACKEND_URL = "http://localhost:8000"
python -m streamlit run frontend/streamlit_app.py

Open http://localhost:8501, then:

1. Upload examples/example_openapi.json and click Analyze Upload.
2. Open Target API. The base URL is pre-filled. Run "Run all tests" once without a token to see the 401 explanation.
3. Enter the token secret, save, and run "Run all tests" again. It finds the two seeded bugs.

Docker Compose files (docker-compose.yml) are included but have not been build-tested. From inside a container, use host.docker.internal instead of localhost for the target URL.

Agent tools

Tool	Purpose
list_apis	Show the endpoints of the uploaded spec
run_tests(category, endpoint_filter)	Run the test library and save results
get_results(run_id)	Show a saved run (latest if empty)
create_defects(run_id)	Create defect records from a run's failures
get_failure_trends	Which endpoints fail most across runs

The same functions are also exposed as an MCP server (python -m mcp_tools.server) for other MCP clients. The built-in agent calls them in-process.

HTTP API

Endpoint	Purpose
POST /upload	Upload a spec; returns project id, detected format, target hints
GET/PUT /projects/{id}/target	Read or set the base URL and token (the token is never returned)
POST /agent/chat	Send a message to the agent
GET /projects, GET /projects/{id}/apis	List projects and their canonical APIs
GET /history, GET /reports	Run history and failure trends

Configuration

Environment variables (prefix APP_, or a .env file). See .env.example.

Variable	Default
APP_OLLAMA_BASE_URL	http://host.docker.internal:11434
APP_OLLAMA_MODEL	llama3.2:3b
APP_OLLAMA_NUM_CTX	4096
APP_DATABASE_URL	sqlite+aiosqlite:///./data/api_testing.db

Project layout

ingest/        API-definition adapters -> canonical model (OpenAPI 3.x / Swagger 2.0)
agent/         LangGraph agent and prompt
mcp_tools/     Tool functions shared by the agent and the MCP server
test_library/  Pytest categories run against the target
backend/       FastAPI app
frontend/      Streamlit UI
database/      SQLAlchemy models and session
examples/      Sample specs and a sample target API with seeded bugs
tests/         Unit tests (pytest)

Tests

pytest

tests/ covers the ingestion layer (OpenAPI 3, Swagger 2.0, nested and circular $ref, error cases). The test library itself is run on demand against a live API.

Status and limitations

This is a working prototype, not a finished product. Known gaps:

• Formats: only OpenAPI 3.x / Swagger 2.0. Postman, HAR and cURL adapters are not built.
• Extraction: allOf/oneOf/anyOf, path/query/header parameters, external $refs and per-endpoint security are not handled. Path parameters use the placeholder id 1, which causes false failures on real APIs.
• Auth: static token only. There is no login-flow option, so expiring tokens must be re-entered.
• Test coverage: three categories. No boundary, authorization, workflow or security tests yet.
• Model: a 3B model is not fully reliable at tool selection. There is no evaluation harness yet, and the agent keeps no conversation memory between messages.
• Storage: tokens are stored in plain text in the local SQLite database.
• Deployment: Docker Compose is untested; there is no CI.

Roadmap

1. Test data support (real path-parameter ids) and allOf handling.
2. Postman-collection adapter.
3. Login-flow authentication and multi-user authorization tests.
4. Failure explanation and business-rule tests generated by the LLM.
5. Evaluation harness for tool-selection accuracy across models.