Skip to content

Six documents say the conformance suite has five checks; it has thirteen, and two name checks that do not exist #152

Description

@macanderson

The problem

The conformance suite runs 13 checks. Six documents say it runs five or seven,
one document names a check that does not exist, and one page's whole structure
is built around the wrong number.

contextgraph-conformance/src/lib.rs declares 13 CHECK_* constants and
run_conformance emits all 13. This repository's own committed report agrees:

$ python3 -c "import json;d=json.load(open('registry/contextgraph-example-docs.report.json'));print(len(d['checks']),[c['name'] for c in d['checks']])"
13 ['handshake', 'consent-scope', 'frame-validity', 'verify-honesty', 'budget-honesty',
    'as-of-temporal', 'kinds-filter', 'anchor-relevance', 'provenance-fixture-consistency',
    'shutdown-clean', 'malformed-input-tolerance', 'embedding-fingerprint', 'correlation']

Where it says otherwise

$ rg -n "five checks|5 checks|seven checks|7 checks" --glob '!CHANGELOG.md' .
sdk/README.md:15:| TypeScript | … | ✅ conformant — passes all 7 checks in CI |
sdk/README.md:16:| Python     | … | ✅ conformant — passes all 7 checks in CI |
sdk/README.md:17:| Go         | … | ✅ conformant — passes all 7 checks in CI |
sdk/README.md:26:`conformance-external.sh` asserts the provider is **green** (all seven checks).
README.md:76:| **Conformance** | … | `contextgraph-conformance`, 5 checks |
docs/overview.md:65:| **Conformance** | … | `contextgraph-conformance`, 5 checks |
docs/running-conformance.md:8:## The five checks
docs/protocol-advantages.md:94:| **Conformance verification** | … | `contextgraph-conformance` — 5 checks that deliberately trip each failure mode |

Three other files already say 13 (docs/registry.md, docs/reference-providers.md,
docs/composition-walkthrough.md), so the tree contradicts itself.

docs/running-conformance.md is the worst of them

It is not one stale number — the page is organised around it:

  • The heading is ## The five checks, followed by a five-row table listing
    handshake, frame-validity, budget-honesty, shutdown-clean,
    malformed-input-tolerance. Eight checks are missing, including
    consent-scope, verify-honesty, kinds-filter, anchor-relevance and
    correlation.
  • Its sample transcript ends CONFORMANT — 5 passed, 0 skipped. A real run of
    the bundled fixture prints 13.
  • Its ProviderTarget section says a stdio target runs "all five checks".
  • Its own closing paragraph discusses provenance-fixture-consistency and the
    malformed-digest/stale-digest misbehaviour modes — none of which appear in
    its own table. The page contradicts itself within one screen.
  • Its sample handshake evidence line prints
    query kinds=["doc"], upsert=false, graph=false. describe_handshake in
    contextgraph-conformance/src/lib.rs formats
    "… query kinds={:?}, graph={}" — there is no upsert= segment, because
    ADR 0004 deleted Capabilities::upsert.

docs/protocol-surface.md names a check that does not exist

Its UR1 row cites:

contextgraph-host::FanOut::usage_report; usage-report conformance check

FanOut::usage_report exists. There is no check named usage-report — not
among the 13 provider CHECK_* constants, the 9 HCHECK_* host constants, or
the 4 CCHECK_* composition constants. rg 'usage-report' over the tree
returns only prose.

This is the same shape #22 already recorded for §G2 ("Verified by
frame-validity" with no G2 implementation) — a "Verified by" cell naming a
check nobody wrote. That one has since been fixed in the code; this one has not
been noticed. Small correction in the same table: row D1 cites "verify
conformance check"; the check is called verify-honesty.

The crate's own module header is also short

contextgraph-conformance/src/lib.rs's //! block introduces "The checks (all
against the frozen contextgraph-types contracts):" and then lists ten.
Missing: correlation, kinds-filter, anchor-relevance — each of which has a
CHECK_* constant and a dedicated --misbehave witness in the same crate.

docs/protocol-advantages.md also names the wrong check for budget honesty

Its guarantees table says budget honesty is enforced by
"Host::query_one_isolated budget audit; frame-validity conformance
check". It is budget-honesty; frame-validity checks score, title, citation
label, representation invariants, timestamps and digest grammar, and never looks
at the budget. The same document gets it right further down ("Conformance
(budget-honesty check)").

README.md and docs/overview.md misattribute --misbehave

Both say "The suite ships a --misbehave mode that trips every check on
purpose." --misbehave is a flag on the fixture provider
contextgraph-example-docs, not on the suite and not on contextgraph-inspect,
which has no such flag. And it does not cover every check: shutdown-clean has
no misbehaviour mode among the Misbehave variants.

What I verified

By reading both sides on origin/main at a01ca64: the 13 CHECK_* constants
and their emission in run_conformance; the 13 names in
registry/contextgraph-example-docs.report.json; each quoted document line; the
absence of any usage-report check name in the source; describe_handshake's
format string; the Misbehave variant list and the absence of a shutdown mode;
and contextgraph-inspect's clap definition having no --misbehave.

Nothing here is inferred.

Why it matters more than a stale number

This project's pitch is that "conformant" is a checked claim. The number of
checks behind that claim is the first thing an evaluator looks for, and the
front page understates it by more than half. Worse, docs/running-conformance.md
is the page someone reads to decide whether the suite covers their concern —
consent scoping, correlation, anchor relevance and verify honesty are all
covered and all absent from its list, so a reader concludes the suite is weaker
than it is.

What "done" looks like

  • Every count in the tree matches the suite. Better than recounting: stop
    writing the number. "the conformance suite" reads the same and never goes
    stale. Where a count genuinely helps, generate it.
  • docs/running-conformance.md's table lists all 13 checks, its sample output
    matches a real run, its ProviderTarget prose is corrected, and the
    upsert=false segment is removed from the sample evidence line.
  • docs/protocol-surface.md's UR1 row names a check that exists — or says
    plainly that UR1 is host-side and has no wire check, which is the honest
    answer if none is planned. D1's citation says verify-honesty.
  • docs/protocol-advantages.md's budget-honesty row names budget-honesty.
  • contextgraph-conformance/src/lib.rs's module header lists every check it
    emits.
  • README.md and docs/overview.md attribute --misbehave to the fixture
    provider and drop "every check", or name the one it does not cover.
  • Worth considering in the same pass: a check that the documented names are a
    subset of the CHECK_*/HCHECK_*/CCHECK_* constants. Two of the errors
    above are invented check names, which a mechanical check catches for free and
    a reader does not.

Constraints

  • registry/contextgraph-example-docs.report.json is the generated artifact and
    is correct; use it as the source of truth rather than recounting by hand.
  • P1: Roadmap: from contextgraph/1.0-draft to a frozen 1.0 with an ecosystem #22 tracks the same class of defect at the SPEC level ("Verified by" cells
    pointing at nothing). This issue is about the prose counts and the two
    invented check names in the docs; the SPEC's own tables were separately
    audited and their named checks all exist.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1This cycle

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions