| Field | Value |
|---|---|
| Document | Canonical production-readiness master plan |
| Status | Active — readiness gate reconciled with implementation through P0-3, P0-1/P0-4, P1-2, P2-6 (docs-only P1-7) |
| Audit date | 2026-09-09 |
| Repo version audited | v0.1.0 / main @ post-M-PR8E + P0-3 + P0-1/P0-4 + P1-2 |
| Method | Claims verified against current code under packages/, apps/, tests/, .github/, and canonical docs |
| Hierarchy | PRD → ARCHITECTURE → SYSTEM_DESIGN → IMPLEMENTATION_PLAN → DECISION_LOG → ROADMAP → this doc (readiness gate) |
Related: ARCHITECTURE.md · SYSTEM_DESIGN.md · DECISION_LOG.md · ROADMAP.md · IMPLEMENTATION_PLAN.md · SECURITY.md
Release reconciliation (2026-10-15): The audit above is historical. The latest
published CLI/Core/adapter release is 0.2.0 (2026-09-10), tagged v0.2.0.
Current source contains additional Unreleased changes; see CHANGELOG.
Target A evidence below distinguishes implemented CI gates from checks awaiting
a fresh clean-PyPI verification. Target B remains incomplete.
Conflict rule: Product intent → PRD. Hard boundaries → ARCHITECTURE. Runtime contracts → SYSTEM_DESIGN. Sequencing → IMPLEMENTATION_PLAN. Decisions → DECISION_LOG. Ship / no-ship readiness and Hosted vs Local CLI gates → this document. If an ADR conflicts with readiness findings, propose a superseding ADR — do not silently rewrite history.
Mutiny v0.1.0 is a credible alpha OSS behavioral fuzz engine: Core oracle, Adapter #1 (OpenAI Agents SDK), CLI (init / run / test), sample project, and a local Hosted lineage UI/API. Unit tests are green (126 passed in this audit). PyPI packages are published.
It is not production-ready as a public or multi-tenant Hosted service. With M-PR8E, customer project_path adapter exec_module is permanently removed from Hosted (410 hosted_customer_execution_removed; MUTINY_ALLOW_PROJECT_EXEC ignored). With P0-3, customer project_path is also filesystem-inert on Hosted (410 hosted_filesystem_access_removed — no resolve/read/write of customer trees or policy files). With M-PR7 + P0-1/P0-4, Bearer auth is required for non-loopback Hosted binds (python -m mutiny_api fails closed without a non-empty MUTINY_API_TOKEN); loopback may omit the token for local demo. With P1-2, Hosted applies in-process (per-process, not distributed) rate limits when auth is configured. With P2-6, operators can mutiny db backup / mutiny db restore Hosted SQLite lineage (local tooling ≠ off-site durability). Residual Hosted risks: attestation is not authorization, and multi-tenant ops remain open. Public multi-tenant Hosted remains out of scope.
Target A — Production Local CLI is the strongest supported surface: local execution, opt-in Hosted sync, and an alpha→beta path per ROADMAP. Remaining Target A gaps are polish/release gating (not Hosted RCE).
Target B — Production Hosted is not met. ADR-019 / M-PR8A–E are implemented (CLI executes customer agents; Hosted observes via ingest). Non-loopback binds require auth; Hosted is filesystem-inert for customer project_path and has in-process rate limits. P2-6 adds operator SQLite backup/restore CLI (not off-site automation). Residual blockers: attestation≠authorization, path/URL allowlisting still a target enum, not multi-tenant. Do not market public multi-tenant Hosted.
| Surface | Evidence |
|---|---|
| Core kernel | packages/mutiny_core — policy, campaign, fitness, mutate, minimize, regress; no FastAPI/React/SQL |
| Adapter #1 | packages/mutiny_openai_agents — OpenAIAgentsAdapter, file loader for .mutiny/adapter.py |
| CLI | packages/mutiny_cli — mutiny init / run / test; PyPI mutiny-ai @ 0.2.0; post-tag fixes remain Unreleased |
| Sample project | examples/openai_support_agent/ with offline scripted model path |
| Hosted API | apps/api — campaigns, SSE, minimize, regressions, projects, policy routes (customer FS/exec removed), ingest /api/ingest/v1/* |
| Hosted UI | apps/web — talks HTTP/SSE to API only (ADR-014) |
| Demo harness | apps/demo_agent — in_process_demo still supported by supervisor |
| CI | .github/workflows/ci.yml — unit + integration + reliability (3.11/3.12), CLI smoke, web build, package build |
| Publish | .github/workflows/publish.yml + docs/PUBLISHING.md |
| Tag | v0.2.0 (published 2026-09-10; registry checked 2026-10-15) |
| Claim | Reality |
|---|---|
| IMPLEMENTATION_PLAN.md marks M2–M5 largely Planned/Partial (2026-08-07) | Adapter #1 + CLI init/run/test + sample path are shipped in 0.1.0 |
| ARCHITECTURE: API owns “rate limits” + “target allowlisting” | P1-2: in-process Hosted rate limits implemented (not distributed). “Allowlist” remains a target enum (in_process_demo | openai_agents), not a filesystem/URL sandbox |
| SYSTEM_DESIGN §20: arbitrary remote hosts blocked; treat customer adapter as untrusted vs control plane | ADR-019 observe-only Target B implemented (M-PR8A–E) — Hosted never exec_modules customer trees; CLI executes + ingest |
| PRD / plan: “Redact secrets in traces” | M-PR3: deterministic redaction on persist/display (mutiny_core.redact) |
docker-compose.yml sets MUTINY_DB_PATH |
M-PR4: resolve_db_path() honors env (compose /app/data/mutiny.sqlite); default remains data/mutiny.sqlite |
| ADR-001 “Hosted first” | Superseded for product priority by ADR-017; M-PR2: CLI default is local; Hosted requires --hosted / --hosted-url |
- Primary usable path: developer machine →
pip install mutiny-ai→mutiny init→mutiny run→mutiny test. - Hosted: observe/lineage for customer runs (
mutiny run --hosted= local exec + ingest). Trustedin_process_demomay still run in-process. Non-loopback requiresMUTINY_API_TOKEN; in-process rate limits apply when auth is configured. Not multi-tenant Production Hosted (Target B DoD incomplete). - MVP limitations (not bugs): one adapter; three policy primitives; single concurrent campaign; SQLite; template/LLM mutation quality varies; refund demo remains the bundled sample (product seeds/mutators are policy-derived as of M-PR6).
Two explicit targets. Do not collapse them.
A release is Production Local CLI ready when a developer can, on their own machine, against an agent they are authorized to test:
- Install
mutiny-aifrom PyPI without monorepouv sync. mutiny initscaffolds valid artifacts.mutiny runcompletes a campaign using Core + Adapter #1 only (Hosted is opt-in via--hosted).- Violations are proven by deterministic policy evaluation on real tool traces (no LLM judge).
- Minimize + save under
.mutiny/tests/;mutiny testFAIL→PASS after a documented agent fix. - Secrets are not written plainly into artifacts by default (or are clearly warned + redacted).
- Safety UX: authorized-use messaging; no silent dependency on a Hosted process.
- CI covers unit + sample offline path; release notes match behavior.
- Docs do not claim multi-framework or multi-tenant Hosted.
Out of Local CLI production scope: public Hosted API, multi-tenant auth, remote worker fleets, non-OpenAI adapters.
A release is Production Hosted ready when, in addition to Local CLI readiness:
- Explicit trust boundary documented and enforced in code (see below).
- Authentication + authorization on mutating and data-exfiltrating endpoints.
- Customer agent code does not execute inside the shared API process (or runs only in a hardened per-tenant sandbox with a written threat model).
- Filesystem access is scoped (no arbitrary
project_path→ host FS write/RCE). - Rate limits, concurrency caps, and abuse controls are real.
- Persistence has backup/restore and migration discipline suitable for durable tenant data.
- Deploy story (compose/Railway/etc.) does not contradict the threat model.
- SECURITY.md and ops runbooks match the actual attack surface.
Current Hosted does not meet this full definition despite M-PR8E / P0-3 / P0-1/P0-4 / P1-2. Treating checkbox attestation + a public URL as “production Hosted” remains incorrect.
┌─────────────────────────────────────────────────────────┐
│ Developer machine (single trust domain) │
│ CLI ──► Core ──► load .mutiny/adapter.py ──► agent │
│ │ │
│ └─► policy.yaml, .mutiny/tests/, optional LLM │
└─────────────────────────────────────────────────────────┘
- Trusted: developer OS user, Core binary from PyPI, policy they wrote.
- Untrusted relative to oracle: model outputs, attack genomes (must not become instructions to Mutiny control logic).
- Intentional: executing the customer adapter in-process is acceptable for Local CLI — same trust as running
pytestagainst their project. - Optional Hosted on same machine: bind to loopback for tokenless demo, or set
MUTINY_API_TOKENfor any non-loopback bind. Customerproject_pathis opaque metadata on Hosted (P0-3) — the API does not mount or write the developer’s project tree. Customer adapter execution stays on the CLI.
┌──────────────┐ auth ┌──────────────────────────┐
│ Browser / CLI│ ─────────────► │ Hosted API (control) │
└──────────────┘ │ - no customer exec │
│ - no customer FS access │
│ - persist lineage │
│ - in-process rate limits│
└────────────┬─────────────┘
│ ingest events / artifacts
┌────────────▼─────────────┐
│ CLI-side execution │
│ (ADR-019 / M-PR8A–E) │
└──────────────────────────┘
Hard rule for Hosted production (ADR-019): arbitrary customer Python must not run in the shared Hosted API process. Verified (M-PR8E): customer openai_agents + project_path create/start/minimize/test paths raise HostedCustomerExecutionRemoved before any load_adapter_factory / exec_module. MUTINY_ALLOW_PROJECT_EXEC cannot restore execution. Trusted in_process_demo remains.
SYSTEM_DESIGN §20 treat customer adapter as untrusted vs the control plane — satisfied for Production Hosted customer execution by ADR-019 → M-PR8E.
Classification: P0 security/data-loss/catastrophic · P1 production blocker · P2 significant debt · P3 polish · P4 future.
MVP limitations are labeled as such, not “bugs.”
| ID | Finding | Evidence | Targets |
|---|---|---|---|
| P0-1 | Unauthenticated Hosted API — any client can create/start campaigns, read traces, write policies, delete regressions | MUTINY_API_TOKEN or python -m mutiny_api fails closed before listen; loopback may omit for local demo; M-PR7 Bearer when set |
Hosted |
| P0-2 | In-process RCE via project_path — Hosted resolves arbitrary paths and exec_modules .mutiny/adapter.py |
resolve_project_root, load_adapter_factory, _make_adapter410; no load_adapter_factory in Hosted API |
Hosted (was fatal if network-exposed) |
| P0-3 | Arbitrary policy file write on server FS via PUT /api/policies/content after path resolve |
save_policy_content in app.pyproject_path is filesystem-inert (410 hosted_filesystem_access_removed); no resolve/read/write of customer trees |
Hosted |
| P0-4 | Public bind without auth — Railway 0.0.0.0 + nixpacks start; compose publishes 8000:8000 |
python -m mutiny_api; 0.0.0.0/::/LAN/public require token; compose requires MUTINY_API_TOKEN |
Hosted |
| ID | Finding | Evidence | Targets |
|---|---|---|---|
| P1-1 | Audit metadata shipped (#32); attestation remains acknowledgment, not authorization | Server UTC attested_at in start events / ingest config; false still rejected; no bearer token stored |
Single-tenant token possession; individual identity and multi-tenant AuthZ remain out of scope |
| P1-2 | Docs/architecture claim rate limits and path/URL allowlisting | Partial: in-process Hosted rate limits shipped (P1-2); path/URL allowlisting still only the target enum | Hosted |
| P1-3 | Secret redaction required by PRD — implemented (M-PR3) | mutiny_core.redact; persist/display wiring |
Both (resolved for common patterns) |
| P1-4 | CLI Hosted-first when api_url reachable — surprises operators; pushes project_path to API |
run_cmd.py--hosted / --hosted-url |
Local (ops safety); Hosted blast radius |
| P1-5 | MUTINY_DB_PATH in compose ignored; DB path hardcoded — resolved (M-PR4) |
mutiny_api.db.resolve_db_path |
Hosted / Data |
| P1-6 | CI runs unit only — integration + reliability not gated on PR — resolved (M-PR5) | .github/workflows/ci.yml |
Both / OSS |
| P1-7 | IMPLEMENTATION_PLAN.md stale vs 0.1.0 — misleads maintainers on ship status | Milestone table vs CHANGELOG / code | OSS — resolved (this docs pass) |
| ID | Finding | Evidence |
|---|---|---|
| P2-1 | Campaign defaults to refund-oriented seeds when caller omits seeds — resolved (M-PR6) | default_policy_seeds via AttackFocus; boundary_refund_seeds harness-only |
| P2-2 | Template mutator + LLM prompts hardcode refund/issue_refund/amount>200/ord_1001 — resolved (M-PR6) |
Templates/prompts consume AttackFocus tools/thresholds/probes |
| P2-3 | Featherless concrete client lives in Core (port OK; provider-named surface in kernel package) | mutiny_core/llm/featherless.py, ADR-012 |
| P2-4 | publish.yml verify step hardcodes 0.1.0 — resolved (M-PR5); reads each package pyproject.toml |
Was fail/misleading on next version |
| P2-5 | Regressions table has no project_id; filtering joins via campaigns |
db.py schema |
| P2-6 | SQLite backup/export tooling for Hosted data — resolved (P2-6) | mutiny db backup / restore; mutiny_api.backup |
| P2-7 | Empty integrations/ vs ARCHITECTURE diagram mentioning MCP/skills — resolved (#38) |
integrations/README.md documents planned scope; diagram labels integrations as planned |
| P2-8 | No Dependabot/Renovate; no CODEOWNERS | OSS maintainer load |
| P2-9 | CLI help mentions --no-attestation in error text but flag not defined |
run_cmd.py vs main.py |
| P2-10 | ARCHITECTURE “allowed targets localhost-only” not enforced as a filesystem/URL sandbox for Hosted | Constraint remains a target enum + observe-only boundary; not a general path/URL allowlist |
| ID | Finding |
|---|---|
| P3-1 | Demo pin / Iris narrative still prominent in Hosted meta (demo_pin, iris-demo-pin) |
| P3-2 | Default regression name refund_limit_regression |
| P3-3 | CONTRIBUTING/GOOD_FIRST_ISSUES strong; link this readiness doc from hub |
| P3-4 | Web a11y / copy still partially demo-centric in places |
| P3-5 | container-smoke CI |
| ID | Finding |
|---|---|
| P4-1 | Additional framework adapters |
| P4-2 | Postgres, multi-campaign workers, org workspaces |
| P4-3 | MCP / Skills |
| P4-4 | Policy packs library, richer primitives |
| P4-5 | Multi-tenant / org Hosted identity (beyond single-tenant Bearer) |
- Hosted execution model — in-process customer adapter load is fine for Local CLI; forbidden on shared Hosted. ADR-019 Option A implemented (M-PR8A–E).
- Product narrative vs Hosted-first CLI — ADR-017 says customer CLI primary; M-PR2 aligns defaults (local; Hosted via
--hosted). - Search heuristics coupled to refund demo — resolved for defaults (M-PR6 / ADR-020); refund packs remain harness helpers.
- Single-process asyncio + SQLite — acceptable for Local/demo Hosted; not multi-writer SaaS (ADR-005/007 already acknowledge). ADR-019 deliberately avoids Option B workers for Target B.
- Doc drift (IMPLEMENTATION_PLAN) — addressed (P1-7); keep this readiness doc as the ship/no-ship gate.
| Control | Local CLI | Hosted today |
|---|---|---|
| AuthN | OS user | Bearer when configured (MUTINY_API_TOKEN, M-PR7). Loopback may omit for local demo; non-loopback fails closed without token (P0-1/P0-4). Supported entrypoint: python -m mutiny_api |
| AuthZ | Operator discipline | Token possession; timestamped attestation acknowledgment (P1-1 audit metadata shipped), not individual identity |
| Target isolation | Same process as developer agent | M-PR8E + P0-3: customer adapter exec + customer FS access permanently removed from Hosted; trusted in_process_demo only. ADR-019 observe-only implemented |
| Network bind | N/A (CLI) | Deploy templates may bind 0.0.0.0 only with non-empty MUTINY_API_TOKEN |
| Rate limit | N/A (CLI execution unlimited) | In-process token-bucket (P1-2; not distributed). Enabled by default when token set |
| Secret redaction | Pass (M-PR3) | Pass (M-PR3; Hosted persist/SSE) |
| Oracle integrity | Deterministic evaluator (strong) | Same (strong) |
| Open-internet attack proxy | Not implemented as product | Not a proxy; customer RCE path removed from Hosted |
- AuthN (at least API token / session) on all non-health routes.
- AuthZ: project ownership; no cross-tenant reads.
- Execution: ADR-019 Option A (observe-only) implemented via M-PR8A–E. Customer Python runs on CLI; do not claim workers unless a superseding ADR chooses Option B.
- Filesystem: customer
project_pathis opaque on Hosted (P0-3); no server-side project tree access. - Abuse: in-process rate limits shipped (P1-2); campaign concurrency=1; payload size caps remain; distributed limits / multi-replica not claimed.
- Secrets: redact before persist/SSE; never log API keys.
- Threat model doc updated in SECURITY.md when Hosted leaves localhost-only.
- Document that adapter execution is intentional and equivalent to running project code.
- Default
mutiny runshould not silently escalate to Hosted on loopback without clear UX (M-PR2: local default; Hosted via--hosted). - Redact common secret patterns in saved regressions/traces.
- Keep “authorized testing only” messaging; do not add open-internet target adapters without ADR.
- Core does not import
apps.*/ demo FastAPI routes. - OpenAI Agents SDK imports confined to
mutiny_openai_agents. - Web does not evaluate policies (HTTP client only).
TargetAdapterport +AttackFocusderivation from policy tools/rules.- Policy primitives are data-driven (
deny_tool/require_args/forbid_args).
| Location | What is hardcoded | Impact |
|---|---|---|
campaign/config.py boundary_refund_seeds / default_refund_seeds |
ord_1001, dollar amounts, refund_limit |
Demo/harness pack only (ADR-020); Core default is default_policy_seeds |
fitness/__init__.py |
APR/manager premise list removed (#41); cues now derive from policy tools/argument names | Assistant text only; lexical search hint, not a violation oracle |
CLI init POLICY_YAML |
Refund + delete templates | OK as scaffold, not engine hardcoding |
API schemas.RegressionSaveRequest |
Default name refund_limit_regression |
UX bias (P3; out of M-PR6) |
Supervisor DEFAULT_SAMPLE_PROJECT |
examples/openai_support_agent |
Fine for local Hosted default |
Verdict: Oracle is modular; product seed/mutation targeting is policy-derived (M-PR6 / ADR-020). Refund corpora remain as bundled demo/example helpers, not Core defaults.
| Gate | Status |
|---|---|
| Lineage UX / SSE | Works locally; customer lineage via ingest |
| Core correctness via API | Trusted in_process_demo harness; customer campaigns via CLI |
| AuthN/AuthZ | Partial — Bearer + fail-closed non-loopback (M-PR7 / P0-1/P0-4); attestation≠AuthZ (P1-1) |
| Safe customer code execution | Pass (M-PR8E) — customer exec_module removed; CLI executes |
| FS sandbox | Pass (P0-3) — customer project_path filesystem-inert / opaque |
| Rate limits | Partial (P1-2) — in-process limits shipped; not distributed; path/URL allowlist still enum |
| Durable multi-user data | Partial (SQLite + operator backup/restore CLI — P2-6; not multi-tenant / off-site automation) |
| Internet deploy | Not Target B — token + observe-only reduce blast radius; still single-tenant / no durable ops |
Operating rule for Hosted deploys: Prefer loopback / single-operator. Non-loopback requires MUTINY_API_TOKEN. Do not market Railway/public URLs as multi-tenant production. Customer adapter exec is removed (M-PR8E); customer FS access removed (P0-3); MUTINY_ALLOW_PROJECT_EXEC has no effect. ADR-019 observe-only = CLI exec + Hosted ingest. In-process rate limits apply when auth is configured (P1-2).
| Gate | Status |
|---|---|
init / run / test shipped |
Pass |
| Offline sample path | Pass (MUTINY_SAMPLE_OFFLINE) |
| Local Core path (default) | Pass |
| Default Hosted-first | Fixed (M-PR2) |
| Attestation UX consistency | Pass (BooleanOptionalAction --no-attestation) |
| Domain-general seeds/mutators | Pass (M-PR6 / ADR-020) |
| Secret redaction | Pass (M-PR3) |
| PyPI install story | Pass (mutiny-ai) |
Closest path to Target A: local mutiny run is the production default (M-PR2); harden artifacts; generalize seeds/mutators; keep Hosted optional and opt-in via --hosted.
| Topic | Status |
|---|---|
| SQLite WAL + migrations | Present (SCHEMA_VERSION = "11") |
| Core free of SQL | Pass |
| Env-configurable DB path | Pass (M-PR4) |
| Regressions linked to projects | Indirect via campaign only (P2-5) |
| Backup / restore | Pass (operator CLI) — mutiny db backup / restore (P2-6); not off-site automation |
| Trace size caps | Documented in ARCHITECTURE; enforcement incomplete at API edge |
| Local CLI artifacts | JSON under .mutiny/tests/ — adequate for Target A |
| Topic | Status |
|---|---|
| Unit tests on 3.11/3.12 | Pass (PR gate) |
| Integration tests in CI | Pass — offline tests/integration on PR (M-PR5) |
| Reliability smoke in CI | Pass — tests/reliability on PR (M-PR5) |
| CLI local smoke in CI | Pass — sample mutiny init + mutiny run (no Hosted) |
| Publish workflow | Present; verify reads package pyproject.toml versions (M-PR5) |
| Web typecheck + build in CI | Present (apps/web) |
| Package build in CI | Present (mutiny-core / mutiny-openai-agents / mutiny-ai) |
| Branch protection assumptions | Not verified in this audit (ops) |
| Asset | Status |
|---|---|
| LICENSE MIT, COC, SECURITY, SUPPORT, CONTRIBUTING | Present |
| Issue/PR templates, good-first-issues | Present |
| Docs hub | Present; must point here |
| IMPLEMENTATION_PLAN freshness | Reconciled (P1-7) — Hosted security milestones reflected; future work unchanged |
| Dependabot / CODEOWNERS | Missing (P2-8) |
| Security disclosure path | Present |
| Honest non-goals | Mostly good; Hosted risk under-emphasized |
Current: v0.1.0 (2026-08-09) — initial public promotion.
Do not blindly label next as 0.2.0 / 0.3.0 / 1.0.0 without tying versions to readiness gates.
| Version | Intent | Exit criteria (summary) |
|---|---|---|
| 0.1.x | Alpha patch line | Docs/CI/safety messaging; Hosted security hardening may land without claiming Target B |
| 0.2.0 | Production Local CLI candidate | Target A DoD met; Hosted still labeled secondary / not multi-tenant Production |
| 0.3.0 | Local CLI polish + Hosted safe-mode | Auth token for non-loopback; Hosted cannot exec/FS customer project_path — substantially landed (M-PR7/8E, P0-1/3/4) pending release tag |
| 0.4.0+ | Hosted observe-only lineage | ADR-019 / M-PR8A–E implemented in code; version tag may follow |
| 1.0.0 | Production Hosted + stable contracts | Full Target B DoD + stable public API/CLI contracts (ROADMAP v1 class) |
Patch releases (0.2.1, …) for fixes within a gate. Breaking Core/API contracts require minor bump pre-1.0 and ADR.
Protocol: implement one problem at a time using the 13-step protocol in Definition of Done. Never batch unrelated fixes.
Order is dependency-aware. Each milestone is independently testable.
| Goal | Align docs with reality; warn operators Hosted is localhost-only |
| Problems solved | P1-7; reduces accidental public exposure (process) |
| Files | IMPLEMENTATION_PLAN.md, SECURITY.md, README Hosted section, this doc |
| Architecture impact | None |
| Dependencies | None |
| Tests | Docs-only |
| DoD | Ship status table matches code; SECURITY states Hosted not internet-safe |
| Rollback | Revert docs commit |
| Release | 0.1.x |
| Goal | Prevent accidental RCE on shared deploys without full redesign |
| Problems solved | P0-2/P0-4 blast radius reduction |
| Files | apps/api config (e.g. refuse openai_agents+project_path unless MUTINY_ALLOW_PROJECT_EXEC=1 and/or loopback-only bind docs) |
| Architecture impact | Temporary control; full fix is ADR-019 |
| Dependencies | M-PR0 |
| Tests | Integration: exec disabled by default in “hosted safe” mode |
| DoD | Default production-ish config cannot exec customer adapters |
| Rollback | Feature flag off |
| Release | 0.1.x or 0.2.0 |
| Status | Implemented (historical) — superseded by M-PR8E (ALLOW flag ignored; permanent remove) |
| Goal | mutiny run prefers local Core; Hosted opt-in |
| Problems solved | P1-4, P2-9 |
| Files | mutiny_cli/run_cmd.py, main.py, CLI.md, init mutiny.yaml defaults |
| Architecture impact | Aligns runtime with ADR-017 (product priority) |
| Dependencies | None (can parallel docs) |
| Tests | CLI unit tests for default local / opt-in hosted |
| DoD | Documented default matches ADR-017; Hosted requires explicit flag/url |
| Rollback | Revert CLI default |
| Release | 0.2.0 |
| Status | Implemented — default local; --hosted / --hosted-url opt-in; no silent Hosted fallback; --no-hosted kept as local alias |
| Goal | Meet PRD secret redaction bar for Target A |
| Problems solved | P1-3 |
| Files | mutiny_core (pure redact helpers), CLI save path, API persist path |
| Architecture impact | None if pure function in Core |
| Dependencies | None |
| Tests | Unit: key-shaped strings redacted; regression golden |
| DoD | Saved artifacts and Hosted traces redact configured patterns |
| Rollback | Flag to disable (MUTINY_DISABLE_SECRET_REDACTION=1) or revert |
| Release | 0.2.0 |
| Status | Implemented — mutiny_core.redact; wired at campaign event dumps + SQLite/API evidence + CLI test evidence |
| Goal | Honor MUTINY_DB_PATH; document backup |
| Problems solved | P1-5, start P2-6 |
| Files | mutiny_api/main.py, compose docs |
| Architecture impact | None |
| Dependencies | None |
| Tests | Unit/integration with temp DB path env |
| DoD | Env controls DB location |
| Rollback | Revert main.py |
| Release | 0.2.0 |
| Status | Implemented — resolve_db_path (explicit → MUTINY_DB_PATH → data/mutiny.sqlite); fail-closed on invalid/empty path; parent dirs created. P2-6 completes operator backup/restore via mutiny db |
| Goal | Gate PRs on unit + offline integration (and optional smoke) |
| Problems solved | P1-6; P2-4 |
| Files | .github/workflows/ci.yml, .github/workflows/publish.yml |
| Architecture impact | None |
| Dependencies | Stable offline sample |
| Tests | Workflow itself |
| DoD | tests/integration (offline) runs on PR; documented |
| Rollback | Revert workflow |
| Release | 0.2.0 |
| Status | Implemented — PR CI: backend-tests (unit+integration+reliability × 3.11/3.12), cli-smoke (local sample), web-build, package-build; publish verify uses pyproject.toml versions |
| Goal | Remove refund-only hardcoding from Core search defaults |
| Problems solved | P2-1, P2-2 |
| Files | campaign/config.py, campaign/engine.py, mutate/* |
| Architecture impact | ADR-020 accepted — policy-derived seeds/mutators |
| Dependencies | None beyond Core |
| Tests | Unit: non-refund policy gets non-refund seeds; mutator uses focus tools |
| DoD | Defaults derive from PolicySet/AttackFocus; refund templates only as examples — done (M-PR6) |
| Rollback | Callers may still pass boundary_refund_seeds / default_refund_seeds explicitly |
| Release | 0.2.0 |
| Goal | Minimum auth before any non-loopback Hosted |
| Problems solved | P0-1, P1-1 (partial) |
| Files | apps/api, web middleware headers, CLI Hosted client, SECURITY.md |
| Architecture impact | ADR-021 accepted — single-tenant Bearer token |
| Dependencies | M-PR1 |
| Tests | Integration: 401 without/invalid token; authenticated success; SSE; CLI; M-PR1 preserved |
| DoD | Protected routes require token when MUTINY_API_TOKEN is set — done (M-PR7) |
| Rollback | Unset MUTINY_API_TOKEN for local unauthenticated demo |
| Release | 0.3.0 |
| Status | Implemented — Bearer via MUTINY_API_TOKEN; public health/meta; P0-1/P0-4 fail-closed non-loopback binds via python -m mutiny_api; auth does not restore customer exec/FS |
| Goal | Customer Python never in shared API process; Hosted ingests lineage from CLI |
| Problems solved | P0-2, P0-3 (re-shaped), Target B foundation |
| Files | API supervisor contracts, CLI event/artifact upload, possibly Web “Run” UX |
| Architecture impact | ADR-019 accepted (Option A) — implement observe-only; workers (Option B) rejected for now |
| Dependencies | ADR-019 accepted; M-PR1/M-PR7 |
| Tests | Contract tests: API without customer exec_module; CLI → Hosted ingest path |
| DoD | Threat model satisfied for single-tenant observe-only Hosted |
| Rollback | Safe-mode only Hosted (M-PR1) |
| Release | 0.4.0 → leads to 1.0.0 |
| Status | Done: M-PR8A–E in HOSTED_INGESTION.md (contract + ingest API + CLI sync + Web observe + production customer exec_module removed) |
| Stage | Deliverable | Status |
|---|---|---|
| M-PR8A | CLI → Hosted ingestion contract (identity, events, artifacts, redaction, idempotency, API/CLI semantics) | Done (docs) |
| M-PR8B | Hosted /api/ingest/v1/* + persistence + contract tests |
Done (server) |
| M-PR8C | CLI local exec + sync for --hosted |
Done (CLI) |
| M-PR8D | Web observe copy (SSE already published from ingest) | Done (Web) |
| M-PR8E | Remove/disable Production Hosted customer project_path exec_module |
Done |
Scheduled after Target A gate; do not block 0.2.0.
- Fresh clean-PyPI
pip install mutiny-ai==0.2.0+mutiny --helprecheck: publication verified 2026-10-15; this docs pass did not repeat isolated install - Sample
mutiny runcompletes locally and saves a violation (workspace smoke verified 2026-10-14; isolated wheel path is gated by release CI) - Regression FAIL to PASS procedure documented in the sample README;
scripts/verify_release_artifacts.shgates exit 1 before the enforcement switch and exit 0 after it on isolated built wheels (reviewed 2026-10-15) - No Hosted required; CLI defaults local and Hosted sync is opt-in
- Secret redaction tests green
- Seeds/mutators policy-general (M-PR6 / ADR-020); refund packs remain demo/harness helpers
- CI: unit + offline integration green on PR
- README/SECURITY/IMPLEMENTATION_PLAN match code (P1-7 docs reconcile)
-
v0.2.0tagged with CHANGELOG; all three 0.2.0 distributions published 2026-09-10 (registry/tag verified 2026-10-15)
- All Target A checks
- AuthN on mutating routes when
MUTINY_API_TOKENset (M-PR7); non-loopback requires token (P0-1/P0-4) - No customer
exec_modulein API process for Production Hosted (ADR-019 → M-PR8E) - Customer
project_pathfilesystem-inert / opaque (P0-3) - In-process rate limits enforced (P1-2) — distributed limits not claimed
- DB backup/export procedure (P2-6 —
mutiny db backup/restore);MUTINY_DB_PATHconfigurable (done M-PR4) - Threat model in SECURITY.md matches observe-only + bind/auth + rate limits
- Version 1.0.0 (or explicit 0.4+ “Hosted beta” with same technical bar, labeled beta)
- P1-1 audit metadata: server UTC attestation timestamp persisted for starts and ingest; retries retain original ingest timestamp; no raw bearer token stored (#32).
- Multi-tenant AuthZ / durable ops as needed for full Target B; timestamped acknowledgment is not authorization.
Never batch unrelated fixes. For each milestone / finding:
- Select exactly one problem ID (e.g.
P1-5) or milestone (M-PR4). - Re-read this doc + linked ADR/ARCHITECTURE constraints.
- Write a failing test or minimal repro (unless docs-only).
- Implement the smallest change that makes the test pass.
- Avoid drive-by refactors and unrelated files.
- Update docs only if contracts/claims change — never to hide a bug.
- Open/adjust ADR if the change alters an accepted decision.
- Run targeted tests for the change.
- Run
uv run pytest tests/unit -q(and integration if touched). - Commit one focused commit (message explains why).
- Record status against the milestone DoD in this file or IMPLEMENTATION_PLAN.
- Stop and wait for review/approval before starting the next problem.
- Do not start the next P0/P1 until the current one’s DoD is checked.
- Should 0.2.0 flip CLI default to local-only, or keep Hosted-first with louder warnings? → Decided (M-PR2): local default; Hosted via
--hosted/--hosted-url. - Preferred Target B shape: (a) observe-only Hosted (CLI runs agent), (b) per-job containers, (c) WASM/other sandbox? → Decided (ADR-019): (a) observe-only; M-PR8 implements; Option B deferred pending superseding ADR.
- Is single shared API token enough for “authenticated single-tenant,” or is user/session required for 1.0? → Near-term (ADR-021 / M-PR7): shared Bearer token; multi-user later.
- How long may refund-biased seeds remain if documented as known limitation for 0.2.0? → Defaults generalized (M-PR6 / ADR-020); refund packs = harness helpers.
- Should Railway config be removed or hard-gated until M-PR1? → Superseded: M-PR8E + P0-1/P0-4; compose/Railway use
python -m mutiny_apiand require token for non-loopback. - Who owns Hosted vs CLI release trains (same version vs separate)?
- LangGraph / CrewAI / PydanticAI / AutoGen / HTTP adapters (ROADMAP Beta)
- MCP / Skills
- Postgres / K8s / worker queues
- Multi-tenant org model / SSO
- Vector attack memory / LLM-as-judge acceptance
- Web CI, visual regression, Dependabot (after Target A)
- Full policy language expansion beyond three primitives
| Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|
| Public Hosted RCE via adapter load | Low (path removed) | Critical | M-PR8E permanent remove; observe-only ingest; no public multi-tenant claim |
| Operators trust attestation as security | High | High | Docs + M-PR7 auth; P1-1 timestamps acknowledgment, not identity or authorization |
| Doc drift causes wrong roadmap work | Med | Med | P1-7 reconcile; this doc as gate |
| Refund hardcoding → false “engine doesn’t work” on other tools | Med | Med | M-PR6 |
| Secret leakage in SQLite/SSE | Med | High | M-PR3 |
| Publish workflow breaks on next version | Low | Low | Fix verify step with dynamic version (M-PR5) |
| Concurrent campaigns / SQLite corruption under abuse | Low locally / Med exposed | Med | Concurrency=1; auth + in-process rate limits (P1-2) |
Status: Accepted — see DECISION_LOG ADR-019 (Option A: observe/lineage Hosted; CLI executes customer adapters). M-PR8A–E implemented — HOSTED_INGESTION.md.
Current code: Hosted never loads customer .mutiny/adapter.py via load_adapter_factory / exec_module. Customer create/start returns 410 hosted_customer_execution_removed. MUTINY_ALLOW_PROJECT_EXEC ignored. Trusted in_process_demo harness unchanged.
Why Target B needed M-PR8: Even with auth (ADR-021), in-process customer exec_module + project_path was RCE/FS-write risk and conflicted with SYSTEM_DESIGN §20. M-PR8E removes that path.
Options evaluated (decision locked):
| Option | Idea | Outcome |
|---|---|---|
| A | Hosted observe/lineage; CLI executes adapter; uploads events/artifacts | Accepted |
| B | Ephemeral per-campaign worker (container/VM) | Rejected for now (ops + ADR-007); reconsider via new ADR |
| C | Hosted = in_process_demo forever; customer projects CLI-only |
Rejected as permanent product shape; M-PR1 already approximates as interim |
Migration (completed via M-PR8A–E + P0-3): M-PR1 fail-closed → CLI→Hosted ingest / Web observe → remove production in-process customer path → customer project_path filesystem-inert → diagrams/docs describe observe-only primary for customer projects.
Status: Accepted and implemented (M-PR6) — see DECISION_LOG ADR-020.
Current decision: Demo reliability may still pin refund boundary seeds (ADR-016); Core campaign defaults to default_policy_seeds from AttackFocus.
Why it failed for general production CLI: Non-refund policies still searched refund conversations; LLM prompts encoded amount > 200 / issue_refund.
Replacement: Default seeds/mutators derive from AttackFocus + policy constraints; keep refund corpora as example pack / demo pin helper only.
Alternatives: Per-domain seed packs selected by policy target field; user-supplied seed files in mutiny.yaml.
Tradeoffs: Slightly weaker out-of-box demo hit rate vs template smoke unless harness keeps refund pack explicitly (boundary_refund_seeds).
Migration: default_policy_seeds(policy) → engine/CLI/API default → leave boundary_refund_seeds as named helper for harness.
ADR-017 already supersedes ADR-001 for product priority. M-PR2 aligns runtime CLI defaults with ADR-017 (local default; Hosted explicit). ADR-019 further locks Hosted out of customer Python execution for Target B.
| Dimension | Score | Notes |
|---|---|---|
| Local CLI | 8.5 | Default local path (M-PR2); common-secret redaction (M-PR3); policy-general seeds/mutators (M-PR6) |
| Hosted | 5.5 | Observe-only + ingest (M-PR8A–E); FS-inert (P0-3); fail-closed bind/auth (P0-1/P0-4); in-process rate limits (P1-2); not multi-tenant Target B |
| Core | 8.0 | Strong oracle & package boundaries; search heuristics policy-derived (M-PR6) |
| OSS | 8.0 | Strong community files; PR CI covers unit/integration/reliability + smoke (M-PR5); IMPLEMENTATION_PLAN reconciled (P1-7) |
| Packaging | 8.5 | PyPI 0.1.0 real; publish verify reads package metadata (M-PR5) |
| Security | 6.5 | Oracle + redaction; customer Hosted exec/FS removed; bind/auth fail-closed; in-process rate limits; remaining: AuthZ (P1-1), durable ops, multi-tenant |
| Overall | 5.5 | Strong Local CLI alpha; Hosted usable as authenticated observe/lineage single-tenant — not dual-target production |
- P0:
Unauthenticated Hosted + in-process adapter exec + policy write + public bind— mitigated (M-PR8E, P0-3, P0-1/P0-4). - P1 status: P1-1 audit metadata shipped (#32); attestation still does not grant authorization. Path/URL allowlisting remains a target enum (remainder of P1-2). (P1-2 rate limits shipped; P1-3…P1-7 resolved.)
- ADR-019 (accepted, Option A): Hosted observe/lineage; CLI executes customer Python — M-PR8A–E done.
- ADR-020 (accepted, M-PR6): Policy-general seeds/mutators.
- ADR-021 (accepted, M-PR7): Single-tenant Bearer token (+ P0-1/P0-4 bind fail-closed).
- Align CLI defaults with ADR-017 (done via M-PR2).
- Refund seed helpers remain as demo/harness packs; Core/CLI/API defaults use
default_policy_seeds— see Modularity Audit table.
IMPLEMENTATION_PLAN milestone status vs 0.1.0 ship— reconciled (P1-7).ARCHITECTURE rate limits missing— in-process limits (P1-2); allowlisting still enum.SYSTEM_DESIGN trust boundary vs Hosted exec— observe-only implemented.- ADR-001 historical vs ADR-017 + CLI local-default (intentional history).
- PRD secret redaction — addressed by M-PR3.
- compose
MUTINY_DB_PATH— addressed by M-PR4.
- M-PR0 Docs truth + danger labeling
- M-PR1 Hosted kill-switch / loopback posture
- M-PR2 Local CLI default path
- M-PR3 Secret redaction
- M-PR4 DB path hygiene
- M-PR5 CI completeness
- M-PR6 Policy-general seeds/mutators (ADR-020)
- M-PR7 Hosted authN token
- M-PR8 ADR-019 Hosted execution model
- P0-3 / P0-1/P0-4 / P1-2 / P1-7 — FS-inert, bind fail-closed, in-process rate limits, docs reconcile (done)
Then remaining authorization / P2-* / P3/P4 work; 1.0.0 only after Target B DoD.
0.1.x (alpha + Hosted security hardening) → 0.2.0 (Production Local CLI) → tagged recognition of Hosted safe-mode / observe-only → 1.0.0 (full Production Hosted + stable contracts).
See checklists under Definition of Done for Target A and Target B separately.
- Created:
docs/PRODUCTION_READINESS.md(this file) - Updated for consistency pointers / ADR proposals / stale-status notes:
docs/README.md,docs/ARCHITECTURE.md,docs/ROADMAP.md,docs/DECISION_LOG.md,docs/IMPLEMENTATION_PLAN.md,SECURITY.md - M-PR8E: Hosted customer
exec_modulepermanently removed (410 hosted_customer_execution_removed); docs updated across ADR-019 surfaces - M-PR8A (docs):
docs/HOSTED_INGESTION.md(+ pointers) — contract defined - M-PR8C (CLI):
mutiny run --hosted= local Core + end-of-run redacted ingest sync — done - M-PR8D (Web): Hosted UI observe-only copy / Local CLI execution badges — done
- P0-3 / P0-1/P0-4 / P1-2: customer FS-inert; bind/auth fail-closed; in-process rate limits — done in code
- P1-7: this document + IMPLEMENTATION_PLAN / SECURITY / related docs reconciled to current implementation — done (docs-only)
- Versioning: Packages remain
0.1.0until an explicit release tag; do not treat security commits as an automatic 0.2/0.3/1.0 bump.
STOP: P1-7 complete — do not begin P2-* or other milestones in this change set.