Skip to content

A feature-gated test file reports green when the feature is off, so it can protect nothing #117

Description

@macanderson

Problem

Both cross-language vector suites in contextgraph-types are gated at file
scope:

  • contextgraph-types/tests/attestation_vectors.rs#![cfg(feature = "attestation")]
  • contextgraph-types/tests/record_vectors.rs#![cfg(feature = "record-attestation")]

When the feature is off, the file compiles to zero tests and cargo prints
test result: ok. 0 passed. That is a green line for a suite that ran nothing.
Both files exist to pin wire-format bytes, and both are the loudest thing that
should fail if a preimage rule changes — a shape that reports success when it
was not built is the opposite of what they are for.

Observed while verifying PR #114's witness. On a tree without the
record-attestation feature, cargo test -p contextgraph-types --all-features --test record_vectors printed test result: ok. 0 passed rather than failing.

Why the existing mitigation is not enough

PR #114 added a features CI job that runs each feature combination by name, so
today the vectors are exercised in CI. That defends the current file list; it
does not defend the next one. A new gated test file added without a matching CI
line is silently inert, and nothing says so.

Files

  • contextgraph-types/tests/attestation_vectors.rs
  • contextgraph-types/tests/record_vectors.rs
  • contextgraph-types/Cargo.toml (the feature list)
  • .github/workflows/ci.yml (the features job)

Reproduce

git stash push -- contextgraph-types/Cargo.toml   # drop the feature definitions
cargo test -p contextgraph-types --all-features --test record_vectors
# test result: ok. 0 passed  <- green, ran nothing

Options

  1. Keep the file ungated and gate only the test bodies, leaving one always-on
    test that asserts the feature list is as expected. A missing feature then
    fails loudly.
  2. Add a guard that reads contextgraph-types/Cargo.toml's [features] and
    fails if the features CI job does not name every one of them, so the job
    cannot fall behind the crate.
  3. Both. (2) is the durable half — it makes the CI job's coverage checkable
    rather than a thing someone remembers.

Done

A gated vector file that is not built by CI causes a failure somewhere, rather
than a green line reporting zero tests. Whichever option is taken carries a test
of its own that fails when the coverage lapses.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1This cycle

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions