feat(provider-tck): emit a machine-readable conformance report

tools/provider-tck/README.md

@@ -295,6 +295,104 @@ documented.
 The Compose stack starts once per suite and is never restarted. Scenario isolation comes from the
 control API.
 
+## Conformance reports
+
+Set `PROVIDER_TCK_REPORT_DIR` and each suite writes a machine-readable report of its run to
+`<dir>/<configuration>.json`, conforming to the [report schema][report-schema] in the specification.
+
+```console
+$ PROVIDER_TCK_REPORT_DIR=./reports mvn test -Dtest='Flagd*TckTest'
+$ jq '.scenarios | group_by(.outcome) | map({(.[0].outcome): length}) | add' reports/flagd-rpc.json
+{
+  "passed": 28,
+  "not-declared": 1
+}
+```
+
+`-Dprovider.tck.report.dir=...` does the same thing and is often easier to pass through Maven. The
+environment variable is the portable spelling — every language's TCK reads it, so one cross-language
+CI job can set one thing.
+
+It is an environment variable rather than a method on `ProviderTckHarness` so that emitting a report
+is a property of the run and not of the code: CI sets it, a developer running the suite locally does
+not, and no adopter changes a line to publish one. Unset means no report, which is not an error.
+Several suites in one JVM each write their own file, so flagd's two resolvers do not collide.
+
+### What the report is for
+
+The per-scenario list is the load-bearing part. This suite promises that a scenario skipped for an
+undeclared capability is reported as skipped with the reason and *never* as passed — and a promise is
+not a check. The report records the outcome of every scenario individually, so a consumer can verify
+the rule instead of trusting a runner's headline number. Go's runner counts capability-gated skips in
+its **passed** tally, which is exactly the failure mode this makes impossible to hide; Cucumber
+reports skips correctly, and the report is what proves it rather than assuming it.
+
+Every scenario appears exactly once, whatever happened to it. A report that quietly omitted the
+scenarios it did not run would satisfy every rule above and still mislead, because a reader would
+have no way to know how many questions went unasked.
+
+Note that `capabilities` summarises the *optional* contract only. Scenarios carrying no capability
+tag are mandatory and roll up into nothing, so a provider can fail one while every capability reads
+`passed`. Read `scenarios` to decide whether a provider conforms.
+
+A capability you declare that **no scenario actually exercised** is left out of `capabilities`
+entirely rather than reported as `passed`. Reporting a green result for a claim nothing examined is
+the vacuous pass the capability vocabulary exists to eliminate, so the suite says nothing instead.
+That happens two ways:
+
+- Nothing in the suite carries the tag. `@targeting` and `@caching` are reserved — they are in the
+  vocabulary so it stays aligned with the flagd test harness, but no scenario carries them yet.
+- Every scenario carrying it was skipped for a *different* capability you did not declare. Both
+  scenarios in `events.feature` carry `@events` plus one of `@stale` or `@configuration-change`, so
+  declaring `@events` on its own runs neither, and a run that ran neither has demonstrated nothing
+  about `@events`.
+
+Exercising is therefore counted by execution, not by tag presence.
+
+### What identifies a scenario
+
+A scenario entry is identified by `feature`, `name` **and** `example` together. The first two are not
+enough: every row of a Scenario Outline shares one name, and the type-mismatch matrix in
+`errors.feature` is eleven rows. `example` is the row's parameters keyed by its Examples column
+header, verbatim as strings — Gherkin has no types, so `"1"` stays a string.
+
+```console
+$ jq '.scenarios[] | select(.feature == "errors") | .example' reports/flagd-rpc.json
+{
+  "key": "string-flag",
+  "requested": "Boolean",
+  "default": "false"
+}
+...
+```
+
+It is absent for a scenario that did not come from an outline, and present for every row that did —
+including a row skipped for an undeclared capability, since eleven skips sharing a name are exactly
+as ambiguous as eleven failures.
+
+### What identifies a report
+
+`provider.name` is what the provider reports through its own metadata, not the suite name. The suite
+name is chosen to read well in a failure message — `flagd-rpc` — which makes it the *configuration*,
+and it is reported as such. One provider with two materially different modes produces two reports
+that are not interchangeable. It is derived from the suite class name (`FlagdInProcessTckTest` →
+`flagd-in-process`) and can be overridden with `ProviderTckHarness.configuration()`.
+
+`tck.specRevision` and `tck.assetsTree` identify the conformance artifacts that were executed, and
+are baked into the JAR at build time from the properties in this module's POM — the artifacts travel
+in the JAR, the repository they came from does not. They are pinned by hand for now because, unlike
+the Go TCK, this module has no spec submodule to read them from; the artifacts under
+`src/main/resources` are vendored copies. See [Where these artifacts should live](#where-these-artifacts-should-live).
+Both are checkable rather than merely asserted:
+`git rev-parse <specRevision>:specification/assets/provider-tck` must reproduce the tree, and the
+tree must match the files in this module.
+
+`sdk.version` is read from the classpath rather than declared, because the TCK depends on an SDK
+version *range* so that adopting it can never force an upgrade — what a consumer actually ran against
+is only knowable at runtime.
+
+[report-schema]: https://github.com/open-feature/spec/blob/main/specification/assets/provider-tck/report/conformance-report.schema.json
+
 ## Relationship to the flagd test harness
 
 The step vocabulary is inherited from the

tools/provider-tck/pom.xml

@@ -14,6 +14,35 @@
 
     <properties>
         <module-name>${groupId}.providertck</module-name>
+
+        <!--
+            THE CONFORMANCE ARTIFACTS THIS BUILD CARRIES
+            ============================================
+            A conformance report has to say which questions were asked, not only what the answers
+            were, so it records the open-feature/spec commit the packaged Gherkin, flag set and
+            control API came from, plus the git tree ID of that directory. Both are filtered into
+            src/main/resources-filtered/.../provider-tck-build.properties and packaged in the JAR,
+            because the repository they came from is not.
+
+            They are pinned here rather than read from git at build time for one reason: unlike the
+            Go TCK, this module has no spec submodule. The artifacts under src/main/resources are
+            vendored copies — see the NOTE below on where they should live — so there is nothing for
+            the build to interrogate. When the submodule arrives, these two values are generated
+            from it instead and nothing else changes; the properties file keeps its name and the
+            report keeps its shape.
+
+            Both are checkable rather than merely asserted, which is what makes a hand-maintained
+            pin tolerable in the meantime. From a checkout of open-feature/spec:
+
+              git rev-parse <revision>
+              git rev-parse <revision>:specification/assets/provider-tck
+              diff -r <spec>/specification/assets/provider-tck/gherkin src/main/resources/features
+
+            Verified byte for byte against dfa16586 at the time of writing.
+        -->
+        <provider-tck.spec.revision>dfa16586d91ca020ef1b3b82a7c972d833ff8f29</provider-tck.spec.revision>
+        <provider-tck.spec.assets-tree>904aa7d5fd7a856a4f92ace24355bd1987143abc</provider-tck.spec.assets-tree>
+
         <assertj.version>3.27.7</assertj.version>
         <awaitility.version>4.3.0</awaitility.version>
         <jackson-databind.version>2.22.1</jackson-databind.version>
@@ -103,6 +132,29 @@
             <artifactId>cucumber-junit-platform-engine</artifactId>
         </dependency>
 
+        <!--
+            Gherkin parser and message types — already on the classpath as transitive
+            dependencies of cucumber-core, and declared here because ConformanceReportPlugin
+            compiles against them directly.
+
+            The conformance report identifies a scenario by the Examples row it came from, and a
+            compiled Cucumber pickle no longer knows which row that was. Re-parsing the feature
+            source Cucumber publishes is the reliable way back to it. Both versions are managed by
+            cucumber-bom in the parent POM, so this cannot drift from the parser Cucumber itself
+            uses — which it must not, or the report would describe a different document than the
+            one that ran.
+        -->
+        <dependency>
+            <groupId>io.cucumber</groupId>
+            <artifactId>gherkin</artifactId>
+            <!-- version managed by cucumber-bom in parent -->
+        </dependency>
+        <dependency>
+            <groupId>io.cucumber</groupId>
+            <artifactId>messages</artifactId>
+            <!-- version managed by cucumber-bom in parent -->
+        </dependency>
+
         <!-- PicoContainer object factory — injects TckState into every step class -->
         <dependency>
             <groupId>io.cucumber</groupId>
@@ -182,4 +234,23 @@
         </dependency>
     </dependencies>
 
+    <build>
+        <!--
+            The conformance artifacts are copied verbatim; only the generated build-info properties
+            are filtered. Filtering the feature files would be a portability hazard rather than a
+            convenience: a scenario is a byte-for-byte shared definition across four languages, and
+            a stray ${...} in one would silently make this language's copy different.
+        -->
+        <resources>
+            <resource>
+                <directory>src/main/resources</directory>
+                <filtering>false</filtering>
+            </resource>
+            <resource>
+                <directory>src/main/resources-filtered</directory>
+                <filtering>true</filtering>
+            </resource>
+        </resources>
+    </build>
+
 </project>

tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/AbstractProviderTckTest.java

@@ -33,12 +33,19 @@
  * {@code dev.openfeature.contrib.tools.providertck.steps}, which reach the harness through
  * {@link TckRuntime}.
  *
+ * <p>The suite also carries {@link ConformanceReportPlugin}, so an adopter needs no configuration to
+ * publish a machine-readable conformance report: setting {@code PROVIDER_TCK_REPORT_DIR} on a run is
+ * enough, and leaving it unset writes nothing.
+ *
  * @see ProviderTckHarness
+ * @see ConformanceReportPlugin
  */
 @Suite
 @IncludeEngines("cucumber")
 @SelectClasspathResource("features")
-@ConfigurationParameter(key = Constants.PLUGIN_PROPERTY_NAME, value = "summary")
+@ConfigurationParameter(
+        key = Constants.PLUGIN_PROPERTY_NAME,
+        value = "summary," + "dev.openfeature.contrib.tools.providertck.ConformanceReportPlugin")
 @ConfigurationParameter(key = Constants.PARALLEL_EXECUTION_ENABLED_PROPERTY_NAME, value = "false")
 @ConfigurationParameter(key = Constants.EXECUTION_MODE_FEATURE_PROPERTY_NAME, value = "same_thread")
 @ConfigurationParameter(key = Constants.GLUE_PROPERTY_NAME, value = "dev.openfeature.contrib.tools.providertck.steps")