Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .quality/evidence/002-project-graph/quality-map.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ target:
aliases: []
source_refs:
- path: "AGENTS.md"
label: "Agents must not set human ratification fields or accept risk for a human"
label: "Agents must not set human validation fields or accept risk for a human"
- path: "agent-skills/quality/references/_shared/layout.md"
label: "Graph artifact layout and per-command edit ownership"

Expand Down Expand Up @@ -171,7 +171,7 @@ expectations:
source_type: "SOURCE"
source_refs:
- path: "AGENTS.md"
anchor: "Agents must not set human ratification fields or accept risk for a human"
anchor: "Agents must not set human validation fields or accept risk for a human"
category: "compliance"
priority: "UNKNOWN"
evidence:
Expand Down
6 changes: 3 additions & 3 deletions .quality/evidence/003-runtime-observations/quality-map.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ target:
- path: "AGENTS.md"
label: "Independence: producers write facts, Quality reads and evaluates them"
- path: "docs/concepts/trust-boundaries.md"
label: "Proof remains independent and scores are deterministic"
label: "Evidence remains independent and scores are deterministic"
- path: "packages/core/src/observations/quality-observations.schema.json"
label: "Canonical observation manifest schema"

Expand Down Expand Up @@ -371,7 +371,7 @@ expectations:
- path: "AGENTS.md"
anchor: "Independence"
- path: "docs/concepts/trust-boundaries.md"
anchor: "Proof remains independent"
anchor: "Evidence remains independent"
category: "security"
priority: "UNKNOWN"
evidence:
Expand Down Expand Up @@ -540,7 +540,7 @@ expectations:
source_type: "SOURCE"
source_refs:
- path: "AGENTS.md"
anchor: "Agents must not set human ratification fields or accept risk for a human"
anchor: "Agents must not set human validation fields or accept risk for a human"
category: "compliance"
priority: "UNKNOWN"
evidence:
Expand Down
2 changes: 1 addition & 1 deletion .quality/evidence/004-quality-scoring/quality-map.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -170,7 +170,7 @@ expectations:
source_type: "SOURCE"
source_refs:
- path: "AGENTS.md"
anchor: "Agents must not set human ratification fields or accept risk for a human"
anchor: "Agents must not set human validation fields or accept risk for a human"
- path: "docs/concepts/trust-boundaries.md"
anchor: "Human decisions stay human"
category: "compliance"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ expectations:
source_type: "SOURCE"
source_refs:
- path: "AGENTS.md"
anchor: "Agents must not set human ratification fields or accept risk for a human"
anchor: "Agents must not set human validation fields or accept risk for a human"
- path: "README.md"
anchor: "It will pause when a human needs to confirm feature boundaries, priorities, or quality checks"
category: "compliance"
Expand Down
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Quality evaluates evidence independently of the systems that produce it.
- Quality reads and evaluates evidence; it does not create tests or modify
producer artifacts.
- Scores are computed by the deterministic engine, never by an agent or UI.
- Agents must not set human ratification fields or accept risk for a human.
- Agents must not set human validation fields or accept risk for a human.

## Dependency direction

Expand Down
2 changes: 1 addition & 1 deletion ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ repository enforces them:
injected by `apps/explorer`, which exposes only the operations the application
needs. Authoring operations remain explicit engine capabilities and are not
available to the Explorer interface.
- Human ratification fields are contract inputs. Agent workflows may propose
- Human validation fields are contract inputs. Agent workflows may propose
changes but must not promote those fields.
- Local and remote filesystem access stays behind data-source adapters. The
adapters discover and normalize inputs; scoring operates on the normalized
Expand Down
4 changes: 2 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,8 +71,8 @@ for the right reason, so you know it covers what you think it covers.

**Test the silence, not just the success.** The worst failures in this tool are
quiet ones — a result that fails to connect, a value that is dropped rather than
rejected. A test proving something works often will not catch those. A test
proving the tool *complains* when something is wrong usually will.
rejected. A test showing that something works often will not catch those. A test
showing that the tool *complains* when something is wrong usually will.

## Releasing

Expand Down
40 changes: 22 additions & 18 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,9 @@ Then open Claude Code in that repository and give it this prompt:
```

The agent will inventory the repository, propose a small quality graph for the
highest-priority feature, connect the proof that already exists, and run an
initial assessment when runtime results are available. It will pause when a
human needs to confirm feature boundaries, priorities, or quality checks.
highest-priority feature, connect the verification methods that already exist,
and run an initial assessment when runtime results are available. It will pause
when a human needs to confirm feature boundaries, priorities, or quality checks.

After the first pass, inspect the proposed files under `.quality/` and answer
the agent's review questions. Then continue with the next command it recommends,
Expand All @@ -38,7 +38,7 @@ Want to control the first slice? Include it in the prompt:
```

The skill evaluates existing specifications, tests, workflows, reports, and
other evidence; it does not create tests or approve its own proposals. See the
other evidence; it does not create tests or validate its own proposals. See the
[`quality` skill](agent-skills/quality/SKILL.md) for all commands and workflow
details.

Expand All @@ -49,35 +49,39 @@ observation set fetched from a CI workflow run.](docs/assets/quality-explorer-ov

## What Quality does

Quality connects product intent to independent, repeatable proof:
Quality connects product intent to independently produced evidence:

```text
requirements (PRDs)
intent validation
features (specs)
quality checks
proof definitions (tests and other evidence)
runtime observations (did the proof pass?)
↓ verification
├── reasoning-based verification → analysis results ─┐
└── empirical verification → empirical observations ┴→ evidence
```

Requirements define desired outcomes, checks state what must hold, and tests,
workflows, telemetry, or manual records provide proof. Quality maps these
relationships in `.quality/` and keeps missing or weak links visible. Agents can
propose maps; people review structure and accepted risk.
Requirements define desired outcomes, and people validate that features and
checks represent that intent. Verification methods then evaluate whether the
implementation or observed behavior satisfies each check. Reasoning-based
methods produce analysis results; empirical methods produce observations of
behavior. Both can contribute evidence. Quality maps these relationships in
`.quality/` and keeps missing or weak links visible. Agents can propose maps;
people validate intent and decide whether to accept risk.

The engine reports four separate measures:

| Measure | Question it answers |
| --- | --- |
| Quality | Is the current proving evidence passing? |
| Coverage | Does every declared check have proof? |
| Evidence confidence | Is the mapped proof strong enough? |
| Quality | What do the current observed results report? |
| Coverage | Does every declared check have a mapped verification method? |
| Evidence confidence | Are the mapped methods strong enough for the claim? |
| Structure confidence | Are these the right features, checks, and priorities? |

See [the concepts guide](docs/README.md) for the model and trust boundaries.
See the [terminology guide](docs/concepts/terminology.md) for the distinction
between validation, verification, evidence, and observation, and the
[concepts guide](docs/README.md) for the full model and trust boundaries.

## Optional: Use `spec-project` for spec-driven testing

Expand Down
61 changes: 36 additions & 25 deletions agent-skills/quality/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
---
name: quality
description: "Construct, assess, and improve a repository's quality project graph and four-score quality index. Use only when the user explicitly invokes /quality or asks for a quality status, score, index, map, graph, or posture—not for a generic request to improve code quality. Guides new-repository setup, project and feature mapping, saved assessment scopes, proof mapping, runtime assessment, and score-directed improvement while preserving independent scoring and human ratification. Commands: start, status, map-project, map-feature, assess, improve, help."
description: "Construct, assess, and improve a repository's quality project graph and four-score quality index. Use only when the user explicitly invokes /quality or asks for a quality status, score, index, map, graph, or posture—not for a generic request to improve code quality. Guides new-repository setup, project and feature mapping, saved assessment scopes, verification-method mapping, runtime assessment, and score-directed improvement while preserving independent scoring and human validation. Commands: start, status, map-project, map-feature, assess, improve, help."
---

# Quality

Quality answers:

> What must this project deliver, what proves each promise, and what does the current
> proof justify believing?
> What must this project deliver, which methods verify each promise, and what does
> the current evidence justify believing?

It maintains a **quality project graph** and asks the deterministic
`quality-tools` engine to compute a four-score **quality index** from that graph.
Expand All @@ -17,7 +17,7 @@ It never invents a score or treats a score as the goal.
**Before any command, read
[independence](references/_shared/independence.md).** A system cannot establish
trust by grading its own claims. This principle governs every workflow,
especially evidence classification and human ratification.
especially evidence classification and human validation.

## Use one formal vocabulary

Expand All @@ -33,9 +33,19 @@ Use these terms consistently:
| Assessment | Project + whole-project or view scope + optional observation set + observed revision/run, producing one quality index; without an observation set the index carries the three static scores and no Quality score |
| Release candidate | A concrete build, version, commit, or artifact covered by an assessment; not a persistent graph grouping |
| Product | Optional descriptive language for what the project delivers; it has no graph or scoring semantics |
| Intent validation | Human confirmation that features, priorities, and checks represent accepted product intent |
| Verification method | A repeatable or inspectable way to evaluate whether implementation or observed behavior satisfies a check |
| Reasoning-based verification | Evaluation by reasoning about an implementation or model, including review, static analysis, model checking, and formal verification |
| Empirical verification | Evaluation by executing or monitoring a system, recording an observation, and comparing it with a check |
| Evidence | Information from reasoning or evaluated observations that supports or challenges a check |
| Empirical observation | A recorded fact about executed or production behavior |
| Runtime observation | Quality's standardized result record; it can carry an executable reasoning tool's outcome without making the underlying method empirical |
| Proof | A deductive argument under stated assumptions; reserve this term for formal or deductive proof |

Do not introduce a second grouping such as `release area`. Use a view when a
reusable subset of features must be assessed together.
Use **evidence gap** in prose; `proof_gap` is only the legacy literal field name
in the current quality-map contract.

## The quality project graph

Expand All @@ -46,24 +56,24 @@ repository (storage)
└── quality project
└── features
└── quality checks: what must hold
└── proof definitions: what could prove each check
└── runtime observations: did that proof pass?
└── verification methods: how each check is evaluated
└── runtime observations: what happened when executable methods ran?

view ─────────────── selects project features
observation set ──── selects runtime-result sources
```

The graph is represented by checked-in files and the proof artifacts they
The graph is represented by checked-in files and the evidence artifacts they
reference:

| Graph layer | Artifact | Meaning |
| --- | --- | --- |
| Project → features | `.quality/project-map.yaml` | Project identity, feature boundaries, priorities, status, and dependencies |
| Feature → checks → proof | `.quality/evidence/<feature>/quality-map.yaml` | What must hold for one feature and which tests, workflows, telemetry, static checks, or manual records prove it |
| Proof → runtime sources | `.quality/config/observation-sources.yaml` | Where canonical observation files come from |
| Feature → checks → verification methods | `.quality/evidence/<feature>/quality-map.yaml` | What must hold for one feature and which tests, workflows, telemetry, static checks, or manual records evaluate it |
| Executable methods → runtime sources | `.quality/config/observation-sources.yaml` | Where canonical observation files come from |
| Runtime source bundles | `.quality/config/observation-sets.yaml` | Which runtime sources are assessed together |
| Saved assessment scopes | `.quality/config/views.yaml` | Which project-map features are included together |
| Actual proof | Files referenced by `evidence.path` | Tests, workflows, reports, runbooks, telemetry, or other auditable proof |
| Evidence artifacts | Files referenced by `evidence.path` | Tests, workflows, reports, runbooks, telemetry, or other auditable evidence |

IDs, paths, and optional test-case names are the graph's edges. Generated
recommendations and `fix-prompts.md` are engine output, not graph source.
Expand Down Expand Up @@ -101,15 +111,16 @@ artifact names:
brownfield, or already partially mapped.
2. **Map the project.** Construct project → feature structure and choose the
highest-priority feature.
3. **Map each feature.** Define its quality checks and connect existing proof.
4. **Connect runtime.** Arrange for proof producers to publish canonical
3. **Map each feature.** Define its quality checks and connect existing
verification methods.
4. **Connect runtime.** Arrange for evidence producers to publish canonical
`quality-observations.json`, within the producer edit boundary below, then
configure transport-only sources and sets that locate those files.
5. **Assess.** Run `quality-tools` and explain the scores together. Coverage,
evidence confidence, and structure confidence are available from step 3
onward; Quality also needs step 4.
6. **Improve.** Diagnose the weak score, improve the underlying structure,
proof, implementation, or runtime wiring, then assess again.
evidence, implementation, or runtime wiring, then assess again.
7. **Repeat.** Expand feature by feature, highest priority and risk first.

`/quality start` orchestrates steps 1–5 for a new or unmapped repository. The
Expand Down Expand Up @@ -171,10 +182,10 @@ repository's priorities. A saved assessment scope is a separate named view in
| Command | Outcome | Reference |
| --- | --- | --- |
| `start [scope]` | Bootstrap the graph in a new, brownfield, or partially mapped repository | [start](references/start.md) |
| `status [scope]` | Read existing artifacts and recent results; do not run proof/scoring commands or edit files | [status](references/status.md) |
| `status [scope]` | Read existing artifacts and recent results; do not run verification/scoring commands or edit files | [status](references/status.md) |
| `map-project [scope]` | Construct or reconcile project → feature structure | [map-project](references/map-project/index.md) |
| `map-feature <target>` | Construct or improve one feature's checks and proof mappings | [map-feature](references/map-feature/index.md) |
| `assess [scope]` | Refresh and explain the four scores without changing graph source or proof | [assess](references/assess.md) |
| `map-feature <target>` | Construct or improve one feature's checks and verification-method mappings | [map-feature](references/map-feature/index.md) |
| `assess [scope]` | Refresh and explain the four scores without changing graph source or evidence artifacts | [assess](references/assess.md) |
| `improve [scope]` | Diagnose weak scores, improve the underlying system, and reassess | [improve](references/improve/index.md) |
| `help [command]` | Explain commands without executing them | [help](references/help.md) |

Expand All @@ -199,7 +210,7 @@ only when `start` or `map-project` requires it.
- use `improve` when the user asks to raise quality posture or close gaps.
5. State what will be read, generated, or edited before acting.
6. Stop at human gates. An agent may propose structure, priorities, reviewed
checks, or accepted risk, but must never ratify for the owner.
checks, or accepted risk, but must never validate intent for the owner.

## The four scores and the honest improvement lever

Expand All @@ -208,10 +219,10 @@ verdict.

| Score | Question | Improve the underlying condition by |
| --- | --- | --- |
| Coverage | Does every declared check have proof? | Create or map missing proof |
| Evidence confidence | Is the mapped proof strong enough? | Use a stronger appropriate modality or add a meaningful gate |
| Quality | Is current proving evidence passing? | Fix the implementation/proof, stale results, or runtime wiring and rerun it |
| Structure confidence | Are these the right features, checks, and priorities? | Ask a human to correct or ratify proposed structure |
| Coverage | Does every declared check have a mapped verification method? | Create or map a missing method |
| Evidence confidence | Are the mapped methods strong enough? | Use a stronger appropriate modality or add a meaningful gate |
| Quality | Are current observed results passing? | Fix the implementation or method, refresh stale results, or repair runtime wiring and rerun it |
| Structure confidence | Are these the right features, checks, and priorities? | Ask a human to correct or validate proposed structure |

Quality is the only one of the four that needs runtime observations. When no
observation set is configured or its results cannot be acquired, report the other
Expand All @@ -227,7 +238,7 @@ rise.
output. When no structured total exists, enumerate first and calculate the
count mechanically; verify that headings, totals, and lists agree.
- Group every cause under the score it actually affects. Missing or
policy-insufficient proof affects coverage; proof type and gate strength
policy-insufficient verification methods affect coverage; method type and gate strength
affect evidence confidence; runtime outcomes/acquisition/resolution affect
Quality; feature/check/priority provenance and review affect structure
confidence. Do not cite a structure gate as an evidence-confidence cause.
Expand All @@ -240,11 +251,11 @@ rise.
or an authoritative artifact already contains it. Never infer attribution
from Git config, environment metadata, or the operating-system account.

## Relationship to proof producers
## Relationship to evidence producers

Quality maps and judges existing proof. It may hand a concrete proof gap to a
Quality maps and judges existing evidence. It may hand a concrete evidence gap to a
producer such as `/shiplight cover`, `/shiplight create-yaml-tests`, or
`/shiplight create-agent-verification`. Producers create or run tests and other
proof; Quality confirms the resulting artifact, connects it to the graph, and
verification methods; Quality confirms the resulting artifact, connects it to the graph, and
remeasures. The only producer-side exception is the explicitly authorized
mechanical workflow glue defined above.
Loading