feat(provider-tck): emit a machine-readable conformance report

[aepfli]

Stacked on #1830 (feat/provider-tck). Part of open-feature/spec#424; the format this emits is defined by the schema in open-feature/spec#425. The equivalent emitter has landed in Go, and this is the Java one.

Set PROVIDER_TCK_REPORT_DIR and each suite writes <dir>/<configuration>.json. Unset means no report, and that is not an error.

Real output

Run against the flagd testbed, both resolver modes, on this branch:

$ PROVIDER_TCK_REPORT_DIR=./reports mvn -pl providers/flagd test -Dtest='Flagd*TckTest'
...
29 scenarios (28 passed, 1 skipped)
29 scenarios (28 passed, 1 skipped)

$ jq '.scenarios | group_by(.outcome) | map({(.[0].outcome): length}) | add' reports/flagd-rpc.json
{
  "passed": 28,
  "not-declared": 1
}

Identical counts for flagd-in-process.json. The one not-declared scenario in each is A float flag is not silently narrowed to an integer, with the reason requires capability @strict-numeric-typing, which this provider does not declare — which is exactly the gap AbstractFlagdTckTest documents. 28 + 1 = 29, the number Cucumber itself reports.

Both files validate against the Draft 2020-12 schema from open-feature/spec#425 using python jsonschema, including the rule added there that anything which did not pass must carry a reason.

What identifies a scenario entry

feature and name are not enough, and the type-mismatch matrix in errors.feature is the proof: it is a Scenario Outline with eleven Examples rows, so the report carries eleven entries under one name. If one row fails and ten pass, a report keyed on feature and name cannot say which failed, and a consumer building a map from it keeps whichever row it saw last.

Each entry therefore also carries example, the row's parameters keyed by their Examples column header:

$ jq -c '.scenarios[] | select(.name | startswith("Requesting the wrong type")) | .example' reports/flagd-rpc.json
{"key":"string-flag","requested":"Boolean","default":"false"}
{"key":"string-flag","requested":"Integer","default":"1"}
{"key":"string-flag","requested":"Float","default":"0.1"}
{"key":"wrong-flag","requested":"Boolean","default":"false"}
{"key":"boolean-flag","requested":"String","default":"fallback"}
{"key":"boolean-flag","requested":"Integer","default":"1"}
{"key":"boolean-flag","requested":"Float","default":"0.1"}
{"key":"integer-flag","requested":"Boolean","default":"false"}
{"key":"integer-flag","requested":"String","default":"fallback"}
{"key":"float-flag","requested":"Boolean","default":"false"}
{"key":"float-flag","requested":"String","default":"fallback"}

Values are the cell contents verbatim, as strings. Gherkin has no types, so "1" stays the string 1; coercing it would make the report say something the table did not. The field is omitted for a scenario that is not an outline row, and present for every row that is — including a row skipped for an undeclared capability, because eleven skips sharing a name are exactly as ambiguous as eleven failures sharing one.

Why a field and not a naming convention. The parameters are the identity, and they come from the feature file rather than from any runner. Mandating a mangled name instead would put a separator, an ordering and an escaping rule into normative text that four languages have to reproduce byte for byte, with drift invisible until two reports silently fail to line up. That is not hypothetical: before the field existed the implementations had already diverged on precisely this point — Go emitted the bare scenario name for all eleven rows, Python appended its pytest node id, JavaScript its runner's expanded title. None of the three is wrong as a runner's display name; none of them is a shared identity either.

How the row is recovered. Cucumber's TestCaseFinished carries a compiled pickle that no longer knows it came from a table, but TestCase.getLocation() resolves the last of the pickle's AST node ids, which for an outline is the Examples TableRow. ConformanceReportPlugin parses the feature source Cucumber publishes on TestSourceRead — not the file, so a consumer supplying features from elsewhere works unchanged — and maps that line back to its row, with headers from the TableHeader. Parsing uses the Gherkin parser Cucumber already depends on, so the report reads the same document the runner executed; io.cucumber:gherkin and io.cucumber:messages were already on the classpath transitively and are now declared, with versions still managed by cucumber-bom.

Gherkin allows a tag on an individual Examples block, so two rows of one outline can differ in whether the capability gate stops them. Nothing here is keyed by scenario name — one TestCaseFinished event produces one entry, and the row lookup is by (uri, line) — so a gated row cannot take its siblings with it. ConformanceReportPluginTest covers that case directly.

Design

Opt-in through the environment, not through code. Emitting a report is a property of the run and not of the provider: CI asks for one, a developer running the suite locally does not, and no adopter changes a line to publish one. -Dprovider.tck.report.dir=... is accepted as well because it is easier to thread through a Maven invocation, and takes precedence; the environment variable is the portable spelling that every language's TCK reads, so one cross-language CI job can set one thing.

The per-scenario list is the point. Appendix F requires that a scenario skipped for an undeclared capability is reported as skipped with the reason and never as passed. That was a promise, not a check. The report records the outcome of every scenario individually, so a consumer can verify the rule rather than trusting a runner's headline number — Go's runner counts capability-gated skips in its passed tally, which is how a summary ends up saying something false.

Cucumber does not have that defect, and this implementation makes it structurally impossible to acquire: one TestCaseFinished event in, one entry out, with the outcome a pure function of Result.getStatus(). The Go emitter had to be fixed for exactly this — godog did not deliver the capability-skip signal to the after-hook, so every skipped scenario was recorded twice, once correctly and once as passed. ConformanceReportPluginTest asserts the property directly: every scenario appears exactly once, the outcome counts add up to the total, and a skip is not-declared with a reason.

A capability nothing exercised is omitted, not passed. A declared capability used to read passed whether or not anything had examined it — the vacuous pass the capability vocabulary exists to eliminate, arriving through the report rather than through the suite. It reached that state two ways.

@targeting is reserved: it is in the tag vocabulary but no scenario carries it, because asserting that an evaluation context reached the backend needs an echo operation the control API does not have. Declaring it bought a green result for free.

The second route is subtler and was found by the Python implementation. A scenario can carry two capability tags and be skipped for the one the provider did not declare, and counting a capability as exercised because a scenario carried its tag counts that skip. events.feature is exactly this shape — the feature is tagged @events and each of its two scenarios adds @stale or @configuration-change — so a provider declaring @events alone ran neither scenario and was told @events: passed. Exercising is now counted by execution: a scenario contributes to its capabilities only when its outcome is passed or failed.

Such a capability is left out of capabilities entirely. Nothing asked the question, so there is no answer, and a consumer sees the tag is absent rather than a pass it cannot rely on. Omitting is preferred to inventing a fifth outcome, because the four in the schema describe what the provider did and "nothing asked this of the provider" is a fact about the run.

flagd declares every capability but @strict-numeric-typing, so its reports now carry seven entries rather than nine — @targeting and @caching are gone. The second route does not change flagd's reports, because the one undeclared capability is the only tag on its scenario; it is a latent bug there, and a live one for a provider that declares @events without @stale.

Everything else about the rollup is unchanged: undeclared is not-declared with a reason, declared-and-exercised-and-failing is failed with a reason that says how many of how many failed, declared-and-exercised-and-passing is passed. This matches go-sdk-contrib#944.

Four outcomes, not two. not-declared and not-applicable are different statements and collapsing them misrepresents a provider. Java emits passed, failed and not-declared; nothing here is not-applicable, which is the honest answer, since no scenario in this suite is unsatisfiable in Java the way @strict-numeric-typing is in JavaScript.

provider.name is the provider's own metadata name, not the suite name. The suite name is chosen to read well in a failure message — flagd-rpc — which makes it the configuration, and it is reported as such. One provider with two materially different modes therefore produces two reports that are not interchangeable, which is the whole reason flagd needs both. The configuration name is derived from the suite class (FlagdInProcessTckTest → flagd-in-process) so nobody has to declare it, and ProviderTckHarness.configuration() overrides it when the derivation does not read well. The provider's own name is observed when a scenario builds one, and falls back to the configuration name if no scenario ever did — the schema will not take an empty string, and the suite name is more useful than a lie.

sdk.version is read, not declared. The TCK depends on an SDK version range so that adopting it can never force an upgrade, so what a consumer actually ran against is only knowable at runtime. It comes from META-INF/maven/dev.openfeature/sdk/pom.properties, falling back to the package manifest and then to unknown.

tck.specRevision and tck.assetsTree are baked in at build time, by Maven resource filtering into a properties file packaged in the JAR — the conformance artifacts travel in the JAR, the repository they came from does not.

They are pinned in the module POM rather than read from git, and that deserves saying plainly: this branch has no spec submodule to read them from. The artifacts under src/main/resources are vendored copies; sourcing them from the submodule is #1838, which is stacked above this one. So there is nothing for the build to interrogate here, and inventing a git call that would find nothing would be worse than a pin. The pin is checkable rather than merely asserted, which is what makes it tolerable in the meantime — and it was checked: the vendored artifacts are byte-for-byte identical to specification/assets/provider-tck at dfa16586, and git rev-parse dfa16586:specification/assets/provider-tck reproduces the recorded tree. It is also the same revision the Go TCK pins, so the two languages are demonstrably answering the same questions.

backend.controlApi reports http, from a method on ControlApiClient rather than a constant in the report builder, so the value belongs to the thing that knows it.

How this interacts with the rest of the stack

Based on #1830 as asked, which means it does not see the two PRs stacked above it, and both will want a small follow-up when this rebases:

• feat(provider-tck): in-process control path and in-memory/multi-provider self-tests #1837 replaces ControlApiClient with a BackendControl interface and adds an in-process control. controlApi() should become a default method on that interface returning empty, overridden by HttpBackendControl with http and InProcessBackendControl with in-process — a default rather than an abstract method, so no existing implementation breaks for the sake of one string. feat(provider-tck): in-process control path and in-memory/multi-provider self-tests #1837 also renames AbstractProviderTckTest to ProviderTckTest, which is where the plugin is registered, and adds the in-memory and multi-provider self-tests that would exercise the in-process path. The conflicts are the same four files as before these two commits — ProviderTckTest, ProviderTckHarness, HttpBackendControl and TckRuntime — and are mechanical; pom.xml and README.md still merge cleanly despite both branches editing them.
• feat(provider-tck): source the spec artifacts from the open-feature/spec submodule #1838 introduces the spec submodule. The two pinned POM properties should then be generated from git rev-parse HEAD and git rev-parse HEAD:specification/assets/provider-tck in the same step that copies the artifacts, which is what keeps a revision from disagreeing with the artifacts beside it. Nothing else changes: the generated properties file keeps its name and location, and the report keeps its shape.

Verification

• mvn -pl tools/provider-tck verify — green, including checkstyle, PMD, spotbugs and spotless. 15 unit tests.
• mvn -pl providers/flagd test -Dtest='Flagd*TckTest' with PROVIDER_TCK_REPORT_DIR set — 58 scenarios, 0 failures, two reports written.
• Both reports validated against the schema with a Draft 2020-12 validator.
• Outcome counts confirmed to sum to the scenario total in both reports.
• (feature, name, example) confirmed unique across every entry of both reports, where (feature, name) alone is not; asserted as a unit test as well.
• @targeting and @caching confirmed absent from capabilities in both reports; the seven that remain are the ones flagd's scenarios actually ran.

Run on JDK 21 with Docker; the flagd suites need a Docker daemon, so they are not part of the module's own test run.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}