Skip to content

validate-examples.py validates every JSON file in tests/fixtures as a record, so any other fixture breaks CI #126

Description

@macanderson

Problem

schema/validate-examples.py finds its subjects by globbing:

fixtures_dir = ROOT / "tests" / "fixtures"
record_fixtures = sorted(
    p for p in fixtures_dir.glob("*.json") if p.name != ATTESTATION_FIXTURE
)

Every .json in that directory is validated against
contextgraph-lifecycle-record.schema.json, with a hand-maintained exclusion
for the one file that is not a record. So any new JSON placed there fails
the schema CI job with a schema error that says nothing about the real
problem — the file is not a record and was never meant to be one.

It has already forced two workarounds. #93 put its cross-language attestation
vectors in a new tests/vectors/ directory specifically to avoid this, and PR
#114 has to edit this script in the same change that adds
record-hash-vectors.json and record-attestation-key.json beside the
records. The exclusion list grows by hand, one file at a time, and the failure
mode when someone forgets is a confusing error rather than a clear one.

tests/fixtures/README.md already states the rule the code does not encode:
the directory is "the canonical home for the Context Exchange Provider
profile's example records", and "the filename stem is the record_kind".

Files

  • schema/validate-examples.py — the glob and ATTESTATION_FIXTURE
  • tests/fixtures/README.md — the stated convention
  • .github/workflows/ci.yml — the schema job

Reproduce

echo '{"hello": "world"}' > tests/fixtures/scratch.json
python3 schema/validate-examples.py    # fails, blaming the record schema

Options

  1. Select by the stated convention instead of by glob: a fixture is a record
    iff its stem is a known record_kind. Anything else is reported as
    "skipped — not a record kind" rather than validated, and an unknown stem
    is a loud failure naming the convention.
  2. Keep the glob and give each non-record fixture an explicit registry at the
    top of the script, so adding one is a one-line declaration rather than an
    edit to a condition.
  3. Move every non-record fixture out of tests/fixtures/ and make the glob's
    assumption true. This is what Port provenance attestation to the TypeScript, Python, and Go SDKs #93 did unilaterally for its own file.

(1) encodes the README's rule, which is the durable half; the others manage
the symptom.

Constraints

  • The script is also what proves the fixtures are valid at all, so whatever
    replaces the glob must still fail when a real record fixture goes missing —
    check("tests/fixtures holds lifecycle record examples", ...) exists for
    that reason and must keep working.
  • PR feat(contextgraph-types): implement record_hash and RecordAttestation #114 is touching this file. Land after it or coordinate.

Done

Adding a non-record JSON file to tests/fixtures/ either is handled by the
stated convention or fails with a message naming that convention, and a test
covers the guard's own direction — a wrong file is rejected, and a record
fixture that disappears is still caught.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Next cycle

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions