| Field | Value |
|---|---|
| Status | Canonical for package boundaries and constraints |
| Last updated | 2026-09-09 |
| Related | PRD · SYSTEM_DESIGN · DECISION_LOG · PRODUCTION_READINESS |
This document defines how Mutiny is structured and what is allowed to depend on what. It does not narrate product vision (see PRD) or runtime sequences (see SYSTEM_DESIGN).
Mutiny is a behavioral fuzz-testing engine: a small trusted kernel wrapped by thin interfaces.
- The kernel (
packages/mutiny_core) decides what a policy violation is, how campaigns search, and how exploits become regressions. It is framework-independent. - The adapter layer is the only place framework-specific glue lives. Additional adapters (LangGraph, CrewAI, …) do not change Core.
- The CLI (
mutiny init/mutiny run) is the primary developer surface for installing Mutiny into a customer agent project. - The Hosted platform (
apps/api,apps/web) makes campaigns operable and visible (lineage, SSE, persistence). - Adapters bridge Core to targets. Adapter #1: OpenAI Agents SDK → developer’s local agent. Bundled demo agent = example adapter target / harness.
- Planned integrations (Skills, MCP, reusable CI Action) will invoke the same kernel or Hosted API. They must not fork logic. See the placeholder README.
We optimize for:
- Correctness of the acceptance oracle
- Debuggability via traces
- A clean path from customer project → adapter layer → Core campaign
- Future OSS extensibility without rewriting the kernel
We explicitly do not optimize for microservice sprawl, multi-cloud topology, or shipping every framework adapter in current scope (one production-quality adapter: OpenAI Agents SDK).
These principles are binding. New features must not violate them without an ADR in DECISION_LOG.md.
| Principle | Meaning |
|---|---|
| Proof over probability | A shipping claim of “violation found” requires deterministic evaluation on a persisted trace—not an LLM opinion. |
| AI proposes; code proves | Models generate/mutate attack text. Code decides policy outcomes and regression PASS/FAIL. |
| Deterministic acceptance | violated ∈ {0,1} for implemented rules is pure and testable. |
| One Core | Campaign, policy, fitness, mutate, minimize, regress live in one package. No second engine. |
| Adapter-first | Core → Adapter Layer → framework adapter → customer project. Framework quirks never leak into Core (ADR-018). |
| Customer project first | Primary path is Mutiny Core → adapter → user project. Hosted and sample/demo are secondary. |
| Regression by default | A verified exploit should be saveable as a permanent test. Discovery without regression is incomplete. |
| Every exploit becomes a permanent test | Minimized reproductions are first-class artifacts, not chat logs. |
| Human-readable evidence | Tool-call JSON, rule IDs, and lineage must be inspectable (CLI and/or Hosted UI). |
| Small trusted kernel | Core stays free of FastAPI, React, and DB drivers. |
| Safety by construction | Current targets are local projects / in-process / localhost; authorized-use expectations required. |
┌──────────────────────────────────────────────────────────┐
│ Presentation apps/web (Hosted; secondary) │
├──────────────────────────────────────────────────────────┤
│ Application apps/api (HTTP, SSE, DB; secondary) │
│ CLI: mutiny init / mutiny run (primary)│
├──────────────────────────────────────────────────────────┤
│ Domain / Core packages/mutiny_core │
│ (framework-independent engine) │
├──────────────────────────────────────────────────────────┤
│ Adapter Layer TargetAdapter port + implementations │
│ • OpenAI Agents SDK Adapter (#1) │
│ • Future: LangGraph, CrewAI, … │
│ • Example: demo_agent adapter │
├──────────────────────────────────────────────────────────┤
│ Targets Customer agent project │
│ Sample / apps/demo_agent (reference) │
├──────────────────────────────────────────────────────────┤
│ Integrations Planned: Skills / MCP / CI │
└──────────────────────────────────────────────────────────┘
Dependency direction: outer layers may call inward. Core must not import apps, integrations, or framework SDKs.
Local CLI → mutiny_core → Adapter Layer → OpenAI Agents SDK Adapter → Customer project
↘ optional sanitized Hosted ingestion (observe/lineage)
↘ example: demo_agent (reference harness)
↘ future adapters (same port; Core unchanged)
Hosted Web/API → SQLite observational data (ingest + trusted in_process_demo only)
✗ customer project_path exec / filesystem
Planned integrations/* → api (preferred) or mutiny_core (offline replay)
Canonical product hierarchy (never reverse):
Mutiny (behavioral fuzz-testing engine)
→ Adapter Layer
→ OpenAI Agents SDK Adapter (Adapter #1)
→ PydanticAI Adapter (source package; CLI-local)
→ Future adapters (LangGraph, CrewAI, AutoGen, HTTP, …)
→ Customer Project
Owns
- Policy schema + evaluator
- Attack genome models
- Campaign orchestration (in-memory / caller-driven)
- Mutation operators (including LLM client ports, not provider SDKs hard-wired as business logic)
- Fitness scoring
- Minimization
- Regression artifact format + replay pure functions
- Target adapter interface (
TargetAdapter) - Trace / event payload types (not persistence)
Must not contain
- FastAPI / Starlette routes
- React / Next.js
- SQLAlchemy/SQLite session management as a required dependency of domain functions
- UI formatting
- MCP protocol servers
- Framework-specific SDK glue (OpenAI Agents SDK, LangGraph, …) — that belongs in adapter implementations outside Core business rules
Allowed
- Pydantic models
- Pure Python
- Optional protocol/ABC for
LLMClient,TargetAdapter,Clock,IdFactory
Persistence is not Core’s job. The API or CLI persists what Core returns.
Owns (implemented)
mutiny init— generate.mutiny/adapter.py,policy.yaml,mutiny.yamlin the customer projectmutiny run— load adapter + policy + config; drive local Core campaign by default; write regressions;--hosted= local + Hosted ingest sync (M-PR8C)mutiny test— replay saved regressions, report PASS/FAIL/SKIPPED and return an exit statusmutiny --version— report installed CLI distribution metadatamutiny db backup/restore— Hosted SQLite operator commands requiring the source/workspacemutiny-apipackage; not included bypip install mutiny-ai
Must not contain
- Reimplemented policy evaluation, fitness, or mutation logic
The mutiny console entrypoint belongs to the mutiny-ai distribution. Core
remains the engine; optional Hosted sync uploads redacted local results.
Owns
- Loading the developer’s agent via OpenAI Agents SDK
- Mapping SDK tool calls / turns →
AdapterTurnResult/ traces - Session reset/step for campaigns
Must not contain
- Policy evaluation or campaign search logic
Owns
- REST + SSE endpoints
- SQLite schema and repositories
- Mapping DB rows ↔ Core objects
- Campaign task supervision (
asyncio) - Authz attestation checks, rate limits, target allowlisting
- Wiring the trusted
in_process_demoharness (customer adapters execute on Local CLI only) - Operator SQLite backup/restore helpers (
mutiny_api.backup; invoked via CLImutiny db, not HTTP)
Must not contain
- Reimplemented policy evaluation
- Custom fitness math divergent from Core
- Mutation prompt logic copied out of Core
M-PR1 (historical): Kill-switch that default-denied customer Hosted exec_module with optional MUTINY_ALLOW_PROJECT_EXEC=1.
M-PR8E (current): Hosted permanently refuses openai_agents + project_path customer adapter exec_module (410 hosted_customer_execution_removed). MUTINY_ALLOW_PROJECT_EXEC is ignored. Local CLI adapter execution is unchanged. Trusted in_process_demo remains.
P0-3: Hosted customer project_path is filesystem-inert (410 hosted_filesystem_access_removed for policy content/list routes). Hosted never resolves/reads/writes customer project trees; project registration stores opaque path labels. Local CLI FS access is unchanged.
M-PR7 / P0-1 / P0-4 (current): Single-tenant Bearer auth via MUTINY_API_TOKEN (ADR-021). Loopback binds may omit the token for local demo. Non-loopback binds (0.0.0.0, ::, LAN/public) require a non-empty token or python -m mutiny_api fails closed before listen. When set, protected /api/* routes require Authorization: Bearer <token>. Public: /api/health, /api/meta. Auth does not enable customer Hosted execution and is not multi-tenant identity.
P1-2: In-process Hosted API rate limits (token bucket; health / normal / expensive / ingest categories). Enabled by default when MUTINY_API_TOKEN is set. Per-process only — not a distributed limiter. 429 rate_limit_exceeded + Retry-After. See SECURITY.md. Local CLI is unaffected. Target allowlisting remains the target enum, not a filesystem/URL sandbox.
P2-6: Hosted lineage durability via operator CLI mutiny db backup / mutiny db restore (SQLite online backup API; validated integrity + schema version). Not a Hosted Web/FS API. Backups are sensitive; local copies do not replace off-host strategy on ephemeral disks. See SECURITY.md.
ADR-019 (implemented via M-PR8A–E): Hosted role for customer projects is observe/lineage — persist campaigns, SSE, artifacts; customer Python runs on Local CLI. Production Hosted must not execute arbitrary customer .mutiny/adapter.py in the shared API process. Trusted in_process_demo harness may remain. See DECISION_LOG ADR-019.
M-PR8A–E: CLI → Hosted ingest contract + Hosted /api/ingest/v1/* + CLI --hosted local-exec sync + Web observe-only copy + production customer exec removed — HOSTED_INGESTION.md.
Owns
- Campaign UX, evolution graph, exploit workflow, tests page
- SSE consumption and REST calls
- Observe-only product copy: Local CLI executes customer adapters; Hosted visualizes synced lineage; trusted
in_process_demolabeled as demo
Must not contain
- Policy evaluation
- Direct model calls for judging violations
- Browser POSTs to
/api/ingest/v1(CLI is the ingestion client)
Owns
- Deliberately vulnerable support agent used as example project, docs target, and reliability harness
- Mock tools and soft system prompt
- Optional HTTP chat surface used by Hosted sample path
Must not contain
- Mutiny campaign logic
- Policy evaluator (policies are Mutiny’s, not enforced inside the vulnerable demo path)
These paths are roadmap placeholders, not shipped adapter locations. Adapter #1
is in packages/mutiny_openai_agents; integrations/README.md
documents the reserved directory. MCP, published Skills, and a reusable CI Action
are not shipped there; the repository's own CI workflow is separate.
The additional PydanticAI adapter
lives in packages/mutiny_pydantic_ai and is implemented but not yet published.
| Path | Role |
|---|---|
| LangGraph / CrewAI / HTTP adapters | Roadmap (Beta/v1) — see ROADMAP |
integrations/mcp |
MCP tool wrappers around API/Core (roadmap) |
integrations/skills |
Instructional Skill markdown (roadmap) |
Must not contain mutation, policy, or fitness logic.
Checked-in policies, sample regressions, sample OpenAI Agents SDK project configs, demo fixtures. Not the primary shipped product.
Unit tests prefer Core. Integration tests may boot API + sample/demo adapter. Reliability tests enforce smoke flake budgets.
| Module | Responsibility | Side effects |
|---|---|---|
policy |
Parse/validate rules; evaluate traces | None |
genome |
Attack candidate structure | None |
adapter |
TargetAdapter ABC |
I/O only in implementations |
trace |
Trace / PolicyHit models | None |
redact |
Deterministic secret redaction for persist/display copies | None |
fitness |
Score traces given PolicySet | None |
mutate |
Produce child genomes | May call LLMClient port |
campaign |
Generational loop driving adapter + fitness + mutate | Calls ports; emits events via callback |
minimize |
Delta-debug genomes | Re-exec via adapter |
regress |
Serialize/replay regression artifacts | Replay via adapter |
events |
Typed event payloads for SSE / CLI bridging | None |
mutiny_coremust not importapps.*orintegrations.*.apps.webmust not importmutiny_coredirectly in current scope (talk to API only)—keeps browser boundary clean.apps.apiis the only layer that speaks SQL for Hosted.- Integrations / CLI must not import
apps.web. - No circular imports between campaign ↔ mutate ↔ fitness; use unidirectional calls from campaign.
- Provider SDKs (OpenAI/Featherless) and agent-framework SDKs live behind ports / adapter implementations—business rules stay provider- and framework-agnostic.
- New frameworks → new adapter implementations; do not broaden Core for framework quirks. Campaign, policy, minimize, regression, and fitness stay framework-independent.
| Area | Primary owner mindset |
|---|---|
| Policy semantics | Core + ADR if changing acceptance meaning |
| OpenAI Agents SDK adapter (Adapter #1) | Adapter module + tests |
| Future framework adapters | New adapter modules; Core unchanged |
| CLI init/run | CLI package / entrypoint |
| Hosted UX | apps/web |
| Persistence / SSE | apps/api |
| Sample/demo reliability | apps/demo_agent + campaign config (harness only) |
| Future adapters / MCP | integrations/* after Adapter #1 is green |
Cross-cutting changes to acceptance oracles require ADR + Core tests before UI work.
These exist to prevent demo failure and architecture drift. Override only via ADR.
| Constraint | Current default | Hard max |
|---|---|---|
| Concurrent campaigns per process | 1 | 1 |
Population size N |
8 | 12 |
Generations Gmax |
6 | 8 |
| Max user turns per genome | 4 | 6 |
| Max mutation LLM retries | 2 | 3 |
| Candidate execution parallelism | 2 | 3 |
| Constraint | Target / limit |
|---|---|
| Max wall-clock campaign duration | 12 minutes (soft stop); hard cancel at 15 |
| Target turn timeout | 30 s |
| Candidate timeout | 120 s |
| Minimize wall-clock budget | 3 minutes |
| Regression replay timeout per test | 120 s |
| SSE client idle tolerance | Reconnect via REST snapshot |
| Constraint | Limit |
|---|---|
| Max message chars per turn | 4_000 |
| Max trace JSON size per candidate | 512 KB |
| Max events retained per campaign | 10_000 |
| Assumed memory (single machine) | ≤ 2 GB process RSS for local demos |
| Constraint | Limit |
|---|---|
| Max target tokens per turn (completion) | provider-dependent; cap max_tokens ≤ 1024 for target replies |
| Max mutation completion tokens | 1_024 |
| Max campaigns without health check | Refuse start if Hosted /api/health model probe fails |
| Constraint | Rule |
|---|---|
| Allowed targets (current) | Local customer project via adapter; in-process sample/demo; http://127.0.0.1 / localhost only |
| Open internet targets | Forbidden in current scope |
| Real side-effecting tools | Forbidden in demos (mocks); customer projects must attest authorized testing |
| Authorization attestation | Required to start Hosted campaign; CLI documents authorized use |
| Constraint | Rule |
|---|---|
| Supported adapters | OpenAI Agents SDK; PydanticAI source package (unreleased). Both implement the same Core port and execute locally. |
| LangGraph, CrewAI, AutoGen, MCP targets | Out of current scope — ROADMAP |
| Pressure | Response |
|---|---|
| Flaky LLM judging destroys trust | Deterministic Core oracle |
| Demo-as-product confuses buyers | Customer project + adapter is primary; demo is sample |
| Dual engines waste time | One Core; CLI + Hosted clients |
| UI logic drifting into “soft” security claims | Web cannot evaluate policies |
| Persistence coupling blocking tests | Core pure; API owns SQLite |
| Framework sprawl | Hard current-scope lock to OpenAI Agents SDK |
| Future CLI/MCP forks | Clients call Core/API; no logic copy |
Detailed behavior lives in SYSTEM_DESIGN.md.
| Subsystem | One-line purpose |
|---|---|
| Policy Engine | Decide if a trace violates explicit rules |
| Campaign Engine | Search adversarially across generations |
| Mutation Engine | Produce policy-focused child attacks |
| Fitness Engine | Guide search; gate on deterministic violation |
| Trace Engine | Record evidence |
| Minimizer | Shrink reproducing exploits |
| Regression Engine | Permanent PASS/FAIL artifacts |
| OpenAI Agents SDK Adapter | Bridge Core to the developer’s agent |
| CLI init/run | Scaffold and drive campaigns in-project |
| Hosted API | Operate and persist campaigns (secondary) |
| Hosted Web | Make search and proof legible (secondary) |
| Sample / Demo Agent | Reference vulnerable target for docs + harness |
| Future integrations | Additional adapters and clients |
Without a new ADR and a concrete current-scope need:
- Vector databases / RAG “attack memory”
- Kafka / Celery / Redis-required queues
- Kubernetes
- Multi-tenant control planes
- Second policy engine in the agent under test for the vulnerable sample path
- LLM-as-judge acceptance path
- Multiple framework adapters in current scope
- New target types → new adapter implementation; Core interface stable.
- New policy primitives → Core evaluator + unit tests + ADR.
- New Hosted pages → API contracts first.
- New integrations → call existing API/Core; no logic copy.
- Softening sample/demo agent for reliability → document in demo config; never fake violations in DB.
- Customer-project path remains primary even when Hosted still wires the sample/demo reference adapter in code.