From 221aad3f0019445d9bcf4f6f4721b5bfaa032b8c Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Mon, 24 Aug 2026 17:53:09 +0200 Subject: [PATCH 1/3] feat(provider-tck): emit a machine-readable conformance report Set PROVIDER_TCK_REPORT_DIR and each suite writes /.json, conforming to the report schema in the OpenFeature specification. Unset means no report, which is not an error. Emitting a report is a property of the run rather than of the code, which is why it is an environment variable and not a method on ProviderTckHarness: CI asks for one, a developer running the suite locally does not, and no adopter changes a line to publish one. -Dprovider.tck.report.dir does the same thing for a Maven invocation; the environment variable is the portable spelling every language's TCK reads. The per-scenario list is the load-bearing part. This suite promises that a scenario skipped for an undeclared capability is reported as skipped with the reason and never as passed, and a promise is not a check. The report records the outcome of every scenario exactly once, straight from Cucumber's TestCaseFinished event, so a consumer can verify the rule instead of trusting a runner's headline number. The Go TCK shipped a version of this that recorded every skipped scenario twice, once correctly and once as passed, because its capability-skip signal did not reach the after-hook; one event in, one entry out removes that whole class of bug here, and ConformanceReportPluginTest asserts the totals add up. provider.name is what the provider calls itself through its own metadata, not the suite name. The suite name is chosen to read well in a failure message -- flagd-rpc -- which makes it the configuration, and it is reported as such. It is derived from the suite class name and overridable with ProviderTckHarness.configuration(). tck.specRevision and tck.assetsTree identify the conformance artifacts that ran. They are baked into the JAR at build time by Maven resource filtering, because the artifacts travel in the JAR and the repository they came from does not. They are pinned in the module POM for now: unlike the Go TCK this module has no spec submodule to read them from, so there is nothing for the build to interrogate. Both are checkable rather than merely asserted, and the vendored artifacts were verified byte for byte against the revision recorded. sdk.version is read from the classpath rather than declared, since the TCK depends on an SDK version range and what a consumer ran against is only knowable at runtime. Verified against the flagd testbed in both resolver modes: 29 scenarios each, 28 passed and 1 not-declared (@strict-numeric-typing), both reports valid against the Draft 2020-12 schema. Signed-off-by: Simon Schrottner --- tools/provider-tck/README.md | 63 ++++ tools/provider-tck/pom.xml | 48 +++ .../providertck/AbstractProviderTckTest.java | 9 +- .../tools/providertck/ConformanceReport.java | 217 +++++++++++ .../providertck/ConformanceReportPlugin.java | 350 ++++++++++++++++++ .../tools/providertck/ControlApiClient.java | 13 + .../contrib/tools/providertck/Outcome.java | 55 +++ .../tools/providertck/ProviderTckHarness.java | 17 + .../tools/providertck/ReportNames.java | 83 +++++ .../tools/providertck/TckBuildInfo.java | 135 +++++++ .../tools/providertck/TckRunMetadata.java | 82 ++++ .../contrib/tools/providertck/TckRuntime.java | 36 ++ .../providertck/steps/ProviderSteps.java | 12 +- .../providertck/provider-tck-build.properties | 8 + .../ConformanceReportPluginTest.java | 324 ++++++++++++++++ 15 files changed, 1446 insertions(+), 6 deletions(-) create mode 100644 tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ConformanceReport.java create mode 100644 tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ConformanceReportPlugin.java create mode 100644 tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/Outcome.java create mode 100644 tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ReportNames.java create mode 100644 tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/TckBuildInfo.java create mode 100644 tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/TckRunMetadata.java create mode 100644 tools/provider-tck/src/main/resources-filtered/dev/openfeature/contrib/tools/providertck/provider-tck-build.properties create mode 100644 tools/provider-tck/src/test/java/dev/openfeature/contrib/tools/providertck/ConformanceReportPluginTest.java diff --git a/tools/provider-tck/README.md b/tools/provider-tck/README.md index e9d8232c3..0c05c1904 100644 --- a/tools/provider-tck/README.md +++ b/tools/provider-tck/README.md @@ -295,6 +295,69 @@ documented. The Compose stack starts once per suite and is never restarted. Scenario isolation comes from the control API. +## Conformance reports + +Set `PROVIDER_TCK_REPORT_DIR` and each suite writes a machine-readable report of its run to +`/.json`, conforming to the [report schema][report-schema] in the specification. + +```console +$ PROVIDER_TCK_REPORT_DIR=./reports mvn test -Dtest='Flagd*TckTest' +$ jq '.scenarios | group_by(.outcome) | map({(.[0].outcome): length}) | add' reports/flagd-rpc.json +{ + "passed": 28, + "not-declared": 1 +} +``` + +`-Dprovider.tck.report.dir=...` does the same thing and is often easier to pass through Maven. The +environment variable is the portable spelling — every language's TCK reads it, so one cross-language +CI job can set one thing. + +It is an environment variable rather than a method on `ProviderTckHarness` so that emitting a report +is a property of the run and not of the code: CI sets it, a developer running the suite locally does +not, and no adopter changes a line to publish one. Unset means no report, which is not an error. +Several suites in one JVM each write their own file, so flagd's two resolvers do not collide. + +### What the report is for + +The per-scenario list is the load-bearing part. This suite promises that a scenario skipped for an +undeclared capability is reported as skipped with the reason and *never* as passed — and a promise is +not a check. The report records the outcome of every scenario individually, so a consumer can verify +the rule instead of trusting a runner's headline number. Go's runner counts capability-gated skips in +its **passed** tally, which is exactly the failure mode this makes impossible to hide; Cucumber +reports skips correctly, and the report is what proves it rather than assuming it. + +Every scenario appears exactly once, whatever happened to it. A report that quietly omitted the +scenarios it did not run would satisfy every rule above and still mislead, because a reader would +have no way to know how many questions went unasked. + +Note that `capabilities` summarises the *optional* contract only. Scenarios carrying no capability +tag are mandatory and roll up into nothing, so a provider can fail one while every capability reads +`passed`. Read `scenarios` to decide whether a provider conforms. + +### What identifies a report + +`provider.name` is what the provider reports through its own metadata, not the suite name. The suite +name is chosen to read well in a failure message — `flagd-rpc` — which makes it the *configuration*, +and it is reported as such. One provider with two materially different modes produces two reports +that are not interchangeable. It is derived from the suite class name (`FlagdInProcessTckTest` → +`flagd-in-process`) and can be overridden with `ProviderTckHarness.configuration()`. + +`tck.specRevision` and `tck.assetsTree` identify the conformance artifacts that were executed, and +are baked into the JAR at build time from the properties in this module's POM — the artifacts travel +in the JAR, the repository they came from does not. They are pinned by hand for now because, unlike +the Go TCK, this module has no spec submodule to read them from; the artifacts under +`src/main/resources` are vendored copies. See [Where these artifacts should live](#where-these-artifacts-should-live). +Both are checkable rather than merely asserted: +`git rev-parse :specification/assets/provider-tck` must reproduce the tree, and the +tree must match the files in this module. + +`sdk.version` is read from the classpath rather than declared, because the TCK depends on an SDK +version *range* so that adopting it can never force an upgrade — what a consumer actually ran against +is only knowable at runtime. + +[report-schema]: https://github.com/open-feature/spec/blob/main/specification/assets/provider-tck/report/conformance-report.schema.json + ## Relationship to the flagd test harness The step vocabulary is inherited from the diff --git a/tools/provider-tck/pom.xml b/tools/provider-tck/pom.xml index 3de834808..7633ff37b 100644 --- a/tools/provider-tck/pom.xml +++ b/tools/provider-tck/pom.xml @@ -14,6 +14,35 @@ ${groupId}.providertck + + + dfa16586d91ca020ef1b3b82a7c972d833ff8f29 + 904aa7d5fd7a856a4f92ace24355bd1987143abc + 3.27.7 4.3.0 2.22.1 @@ -182,4 +211,23 @@ + + + + + src/main/resources + false + + + src/main/resources-filtered + true + + + + diff --git a/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/AbstractProviderTckTest.java b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/AbstractProviderTckTest.java index 4f3304d94..bd214c990 100644 --- a/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/AbstractProviderTckTest.java +++ b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/AbstractProviderTckTest.java @@ -33,12 +33,19 @@ * {@code dev.openfeature.contrib.tools.providertck.steps}, which reach the harness through * {@link TckRuntime}. * + *

The suite also carries {@link ConformanceReportPlugin}, so an adopter needs no configuration to + * publish a machine-readable conformance report: setting {@code PROVIDER_TCK_REPORT_DIR} on a run is + * enough, and leaving it unset writes nothing. + * * @see ProviderTckHarness + * @see ConformanceReportPlugin */ @Suite @IncludeEngines("cucumber") @SelectClasspathResource("features") -@ConfigurationParameter(key = Constants.PLUGIN_PROPERTY_NAME, value = "summary") +@ConfigurationParameter( + key = Constants.PLUGIN_PROPERTY_NAME, + value = "summary," + "dev.openfeature.contrib.tools.providertck.ConformanceReportPlugin") @ConfigurationParameter(key = Constants.PARALLEL_EXECUTION_ENABLED_PROPERTY_NAME, value = "false") @ConfigurationParameter(key = Constants.EXECUTION_MODE_FEATURE_PROPERTY_NAME, value = "same_thread") @ConfigurationParameter(key = Constants.GLUE_PROPERTY_NAME, value = "dev.openfeature.contrib.tools.providertck.steps") diff --git a/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ConformanceReport.java b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ConformanceReport.java new file mode 100644 index 000000000..11ce4a943 --- /dev/null +++ b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ConformanceReport.java @@ -0,0 +1,217 @@ +package dev.openfeature.contrib.tools.providertck; + +import com.fasterxml.jackson.annotation.JsonInclude; +import java.util.List; +import java.util.Map; + +/** + * One run of the conformance suite against one provider in one configuration. + * + *

The field names and nesting are fixed by the report schema in the OpenFeature specification + * repository. This class is deliberately a transcription of that schema rather than a shape that + * would be convenient in Java, because the point of the format is that every language's TCK emits + * the same document. + * + *

Fields left {@code null} are omitted from the JSON. The schema sets + * {@code additionalProperties: false} throughout, so an unexpected field is a validation failure + * rather than something a consumer ignores. + * + * @see open-feature/spec#424 + */ +@JsonInclude(JsonInclude.Include.NON_NULL) +public final class ConformanceReport { + + /** + * The major version of the report schema this document conforms to. + * + *

An integer as a string, so a consumer can reject a report it does not understand rather + * than guessing at it. + */ + public static final String SCHEMA_VERSION = "1"; + + /** The schema version this document claims. */ + public final String schemaVersion; + + /** What was tested. */ + public final Provider provider; + + /** Which OpenFeature SDK the provider was exercised through. */ + public final Sdk sdk; + + /** What asked the questions, and which questions. */ + public final Tck tck; + + /** What the provider was pointed at. */ + public final Backend backend; + + /** Per-capability outcome, keyed by Gherkin tag including the leading at-sign. */ + public final Map capabilities; + + /** Per-scenario outcome, one entry per scenario in the suite. */ + public final List scenarios; + + ConformanceReport( + Provider provider, + Sdk sdk, + Tck tck, + Backend backend, + Map capabilities, + List scenarios) { + this.schemaVersion = SCHEMA_VERSION; + this.provider = provider; + this.sdk = sdk; + this.tck = tck; + this.backend = backend; + this.capabilities = capabilities; + this.scenarios = scenarios; + } + + /** Identifies the provider under test. */ + @JsonInclude(JsonInclude.Include.NON_NULL) + public static final class Provider { + + /** + * What the provider calls itself, through its own metadata. + * + *

Not the name of the suite. A suite name is chosen to read well in a failure message — + * {@code flagd-rpc} — which makes it the {@link #configuration}, not the identity. + */ + public final String name; + + /** The language the provider is written in, always {@code java} here. */ + public final String language; + + /** + * Which configuration of the provider was tested. + * + *

A provider with more than one materially different mode produces one report per mode, + * and they are not interchangeable: flagd's RPC and in-process resolvers differ in whether + * they emit {@code PROVIDER_STALE}, so a report keyed on the provider name alone would have + * to pick one and misrepresent the other. + */ + public final String configuration; + + Provider(String name, String language, String configuration) { + this.name = name; + this.language = language; + this.configuration = configuration; + } + } + + /** Identifies the OpenFeature SDK the run went through. */ + @JsonInclude(JsonInclude.Include.NON_NULL) + public static final class Sdk { + + /** Maven coordinates of the SDK, without a version. */ + public final String name; + + /** The resolved SDK version. */ + public final String version; + + Sdk(String name, String version) { + this.name = name; + this.version = version; + } + } + + /** Identifies the TCK implementation and the conformance artifacts it executed. */ + @JsonInclude(JsonInclude.Include.NON_NULL) + public static final class Tck { + + /** Which TCK implementation produced this report. */ + public final String implementation; + + /** The version of that implementation. */ + public final String version; + + /** The open-feature/spec commit the executed artifacts came from. */ + public final String specRevision; + + /** + * The git tree object ID of {@code specification/assets/provider-tck} at that revision. + * + *

Carried alongside the commit because it identifies the artifacts rather than the + * commit: it is unchanged by unrelated edits elsewhere in the specification, and + * {@code git rev-parse :specification/assets/provider-tck} must reproduce it, + * so a revision recorded wrongly does not go unnoticed. + */ + public final String assetsTree; + + Tck(String implementation, String version, String specRevision, String assetsTree) { + this.implementation = implementation; + this.version = version; + this.specRevision = specRevision; + this.assetsTree = assetsTree; + } + } + + /** Describes what the provider was pointed at. */ + @JsonInclude(JsonInclude.Include.NON_NULL) + public static final class Backend { + + /** A short human-readable description of the stack under test. */ + public final String description; + + /** + * How the backend was driven, {@code http} or {@code in-process}. + * + *

{@code in-process} is the narrow allowance made for providers with no backend; a report + * claiming it for a provider that has one should be treated with suspicion. + */ + public final String controlApi; + + Backend(String description, String controlApi) { + this.description = description; + this.controlApi = controlApi; + } + } + + /** The outcome of one capability, with the reason it is not simply {@code passed}. */ + @JsonInclude(JsonInclude.Include.NON_NULL) + public static final class CapabilityResult { + + /** The capability's outcome across every scenario that gates on it. */ + public final Outcome state; + + /** Why, in a form someone reading a comparison page can use. */ + public final String reason; + + CapabilityResult(Outcome state, String reason) { + this.state = state; + this.reason = reason; + } + } + + /** The outcome of one scenario. */ + @JsonInclude(JsonInclude.Include.NON_NULL) + public static final class ScenarioResult { + + /** The feature file the scenario came from, without extension. */ + public final String feature; + + /** The scenario name as written in the feature file. */ + public final String name; + + /** The scenario's Gherkin tags, including any inherited from the feature. */ + public final List tags; + + /** What happened. */ + public final Outcome outcome; + + /** For a skip, why it was skipped; for a failure, what failed. */ + public final String reason; + + /** How long the scenario took. */ + public final double durationMs; + + ScenarioResult( + String feature, String name, List tags, Outcome outcome, String reason, double duration) { + this.feature = feature; + this.name = name; + this.tags = tags; + this.outcome = outcome; + this.reason = reason; + this.durationMs = duration; + } + } +} diff --git a/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ConformanceReportPlugin.java b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ConformanceReportPlugin.java new file mode 100644 index 000000000..e3a8125b6 --- /dev/null +++ b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ConformanceReportPlugin.java @@ -0,0 +1,350 @@ +package dev.openfeature.contrib.tools.providertck; + +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.ObjectMapper; +import edu.umd.cs.findbugs.annotations.SuppressFBWarnings; +import io.cucumber.plugin.ConcurrentEventListener; +import io.cucumber.plugin.event.EventPublisher; +import io.cucumber.plugin.event.Result; +import io.cucumber.plugin.event.Status; +import io.cucumber.plugin.event.TestCase; +import io.cucumber.plugin.event.TestCaseFinished; +import io.cucumber.plugin.event.TestRunFinished; +import java.io.IOException; +import java.io.UncheckedIOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.InvalidPathException; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.ArrayList; +import java.util.Collections; +import java.util.Comparator; +import java.util.EnumMap; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.Set; +import java.util.function.Supplier; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * Writes a machine-readable conformance report for the suite that just ran. + * + *

Registered automatically by {@link AbstractProviderTckTest}, so an adopter changes nothing to + * get one. It is opt-in per run: set {@value #REPORT_DIR_ENV} (or the + * {@value #REPORT_DIR_PROPERTY} system property) and each suite writes + * {@code

/.json}. Emitting a report is a property of the run rather than of the + * code — CI asks for one, a developer running the suite locally does not — and unset means no + * report, which is not an error. Several suites in one JVM each write their own file, so flagd's two + * resolvers do not collide. + * + *

Why the per-scenario list is the load-bearing part. Appendix F requires that a + * scenario skipped for an undeclared capability is reported as skipped with the reason and never as + * passed. This plugin makes that checkable by a consumer rather than dependent on the runner's + * summary being trustworthy: it records the outcome of every scenario, exactly once, straight from + * Cucumber's own {@code TestCaseFinished} event. One event in, one entry out — there is no path by + * which a skip is also counted as a pass, which is precisely the bug the Go TCK had to fix, where + * the capability-skip signal did not reach the after-hook and every skipped scenario was recorded + * twice. + * + * @see open-feature/spec#424 + */ +public final class ConformanceReportPlugin implements ConcurrentEventListener { + + /** Environment variable naming the directory reports are written to. */ + public static final String REPORT_DIR_ENV = "PROVIDER_TCK_REPORT_DIR"; + + /** + * System property naming the directory reports are written to, taking precedence over the + * environment variable. + * + *

Accepted in addition to {@value #REPORT_DIR_ENV} because {@code -D} is how a Maven or + * Gradle invocation is usually parameterised. The environment variable is the portable spelling + * and is what every other language's TCK reads, so a cross-language CI job can set one thing. + */ + public static final String REPORT_DIR_PROPERTY = "provider.tck.report.dir"; + + private static final Logger log = LoggerFactory.getLogger(ConformanceReportPlugin.class); + + private static final String LANGUAGE = "java"; + + private final List records = Collections.synchronizedList(new ArrayList<>()); + private final Supplier reportDir; + private final Supplier> metadata; + + /** Creates the plugin Cucumber instantiates by name. */ + public ConformanceReportPlugin() { + this(ConformanceReportPlugin::configuredReportDir, TckRuntime::lastRunMetadata); + } + + ConformanceReportPlugin(Supplier reportDir, Supplier> metadata) { + this.reportDir = reportDir; + this.metadata = metadata; + } + + @Override + public void setEventPublisher(EventPublisher publisher) { + publisher.registerHandlerFor(TestCaseFinished.class, this::onTestCaseFinished); + // The end-of-run event carries the run's own result, which the report has no use for: what + // matters is the outcome of each scenario, which TestCaseFinished has already delivered. + publisher.registerHandlerFor(TestRunFinished.class, finished -> writeReport()); + } + + /** + * Returns the directory reports are written to, or {@code null} when none was configured. + * + * @return the configured report directory, trimmed, or {@code null} + */ + static String configuredReportDir() { + String property = System.getProperty(REPORT_DIR_PROPERTY); + if (property != null && !property.trim().isEmpty()) { + return property.trim(); + } + String environment = System.getenv(REPORT_DIR_ENV); + if (environment != null && !environment.trim().isEmpty()) { + return environment.trim(); + } + return null; + } + + private void onTestCaseFinished(TestCaseFinished event) { + TestCase testCase = event.getTestCase(); + Result result = event.getResult(); + records.add(new ScenarioRecord( + featureName(testCase), + testCase.getName(), + Collections.unmodifiableList(new ArrayList<>(testCase.getTags())), + result.getStatus(), + messageOf(result), + result.getDuration().toNanos() / 1_000_000.0)); + } + + private void writeReport() { + String dir = reportDir.get(); + if (dir == null) { + return; + } + + Optional run = metadata.get(); + if (!run.isPresent()) { + // Nothing observed the suite, which means it never got as far as starting the runtime. + // The run has failed for some other reason by now; adding a report with an invented + // provider name on top of that would only mislead. + log.error( + "{} is set but no TCK suite ran, so there is nothing to report on. " + + "This normally means the suite failed before its backend stack started.", + REPORT_DIR_ENV); + return; + } + + write(build(run.get()), dir, run.get().configuration()); + } + + /** + * Assembles the report from what the run observed. + * + * @param run what the runtime recorded about this suite + * @return the report, ready to serialise + */ + ConformanceReport build(TckRunMetadata run) { + List observed; + synchronized (records) { + observed = new ArrayList<>(records); + } + observed.sort(Comparator.comparing((ScenarioRecord r) -> r.feature).thenComparing(r -> r.name)); + + Set declared = run.capabilities(); + List scenarios = new ArrayList<>(observed.size()); + + // Only a capability that something actually gated on can be said to have failed, so this + // counts the failures as the scenarios are converted rather than guessing afterwards. The + // count goes into the capability's reason, which the schema requires for anything that did + // not pass. + Map failed = new EnumMap<>(Capability.class); + + for (ScenarioRecord record : observed) { + scenarios.add(resolve(record, declared, failed)); + } + + return new ConformanceReport( + new ConformanceReport.Provider(run.providerName(), LANGUAGE, run.configuration()), + new ConformanceReport.Sdk(TckBuildInfo.SDK_NAME, TckBuildInfo.sdkVersion()), + new ConformanceReport.Tck( + TckBuildInfo.IMPLEMENTATION, + TckBuildInfo.tckVersion(), + TckBuildInfo.specRevision(), + TckBuildInfo.assetsTree()), + backendOf(run), + capabilitiesOf(declared, failed), + Collections.unmodifiableList(scenarios)); + } + + private static ConformanceReport.ScenarioResult resolve( + ScenarioRecord record, Set declared, Map failed) { + Outcome outcome; + String reason; + + if (record.status == Status.PASSED) { + outcome = Outcome.PASSED; + reason = null; + } else if (record.status == Status.SKIPPED) { + outcome = Outcome.NOT_DECLARED; + reason = skipReason(record, declared); + } else { + outcome = Outcome.FAILED; + reason = record.message == null ? "the scenario was reported as " + record.status : record.message; + for (Capability capability : gatingCapabilities(record.tags)) { + failed.merge(capability, 1, Integer::sum); + } + } + + return new ConformanceReport.ScenarioResult( + record.feature, + record.name, + record.tags.isEmpty() ? null : record.tags, + outcome, + reason, + record.durationMs); + } + + /** + * Explains a skip, preferring the capability the scenario needed over whatever was thrown. + * + *

Deriving the capability from the scenario's own tags rather than from the abort message + * keeps the two from drifting apart, and gives a consistent sentence across every language's + * TCK. The thrown message is the fallback, for a scenario skipped by something other than the + * capability gate. + */ + private static String skipReason(ScenarioRecord record, Set declared) { + for (Capability capability : gatingCapabilities(record.tags)) { + if (!declared.contains(capability)) { + return "requires capability " + capability.tag() + ", which this provider does not declare"; + } + } + return record.message == null ? "the scenario was skipped" : record.message; + } + + private static List gatingCapabilities(List tags) { + List gating = new ArrayList<>(tags.size()); + for (String tag : tags) { + Capability.fromTag(tag).ifPresent(gating::add); + } + return gating; + } + + /** + * Summarises each capability, with the reason the schema requires for anything but a pass. + * + *

This object covers the optional contract only, and is not a verdict on the provider: a + * scenario with no capability tag is mandatory and rolls up into nothing here, so a provider can + * fail one while every entry below reads {@code passed}. The per-scenario list is what a + * consumer has to read to decide whether a provider conforms. + */ + private static Map capabilitiesOf( + Set declared, Map failed) { + Map results = new LinkedHashMap<>(); + for (Capability capability : Capability.values()) { + ConformanceReport.CapabilityResult result; + if (!declared.contains(capability)) { + result = new ConformanceReport.CapabilityResult( + Outcome.NOT_DECLARED, + "not declared by this provider's configuration; the " + capability.tag() + + " scenarios were skipped and did not contribute to this result"); + } else if (failed.containsKey(capability)) { + int count = failed.get(capability); + result = new ConformanceReport.CapabilityResult( + Outcome.FAILED, + count + (count == 1 ? " scenario" : " scenarios") + " carrying " + capability.tag() + + " failed; the per-scenario results say which, and why"); + } else { + result = new ConformanceReport.CapabilityResult(Outcome.PASSED, null); + } + results.put(capability.tag(), result); + } + return Collections.unmodifiableMap(results); + } + + private static ConformanceReport.Backend backendOf(TckRunMetadata run) { + String description = run.backendDescription().orElse(null); + String controlApi = run.controlApi().orElse(null); + return description == null && controlApi == null + ? null + : new ConformanceReport.Backend(description, controlApi); + } + + @SuppressFBWarnings( + value = "PATH_TRAVERSAL_IN", + justification = "The directory is supplied by whoever started the test run, which is the " + + "whole point of the setting; the file name within it is sanitised by ReportNames") + private void write(ConformanceReport report, String dir, String configuration) { + Path directory; + try { + directory = Paths.get(dir); + } catch (InvalidPathException e) { + throw new IllegalStateException( + "provider-tck [" + configuration + "]: " + REPORT_DIR_ENV + " is not a usable path: " + dir, e); + } + Path path = directory.resolve(ReportNames.fileNameOf(configuration)); + + String json; + try { + json = new ObjectMapper().writerWithDefaultPrettyPrinter().writeValueAsString(report) + "\n"; + } catch (JsonProcessingException e) { + throw new IllegalStateException( + "provider-tck [" + configuration + "]: could not encode the conformance report", e); + } + + // A failure to write is raised rather than logged and swallowed. CI that asked for a report + // and silently did not get one is how a publishing pipeline serves a stale result forever. + try { + Files.createDirectories(directory); + Files.write(path, json.getBytes(StandardCharsets.UTF_8)); + } catch (IOException e) { + throw new UncheckedIOException( + "provider-tck [" + configuration + "]: could not write the conformance report to " + path, e); + } + + log.info("provider-tck [{}]: conformance report written to {}", configuration, path); + } + + private static String messageOf(Result result) { + Throwable error = result.getError(); + if (error == null) { + return null; + } + String message = error.getMessage(); + return message == null || message.trim().isEmpty() ? error.toString() : message.trim(); + } + + /** Turns {@code classpath:features/errors.feature} into {@code errors}. */ + private static String featureName(TestCase testCase) { + String uri = testCase.getUri().toString(); + int lastSlash = uri.lastIndexOf('/'); + String base = lastSlash < 0 ? uri : uri.substring(lastSlash + 1); + int extension = base.lastIndexOf('.'); + return extension <= 0 ? base : base.substring(0, extension); + } + + /** One scenario as Cucumber reported it, before it is interpreted against declared capabilities. */ + private static final class ScenarioRecord { + private final String feature; + private final String name; + private final List tags; + private final Status status; + private final String message; + private final double durationMs; + + ScenarioRecord( + String feature, String name, List tags, Status status, String message, double durationMs) { + this.feature = feature; + this.name = name; + this.tags = tags; + this.status = status; + this.message = message; + this.durationMs = durationMs; + } + } +} diff --git a/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ControlApiClient.java b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ControlApiClient.java index 6c0a19e57..527e55876 100644 --- a/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ControlApiClient.java +++ b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ControlApiClient.java @@ -56,6 +56,19 @@ public String baseUrl() { return baseUrl; } + /** + * Returns how the backend was driven, for the conformance report. + * + *

{@code http} is the normative control API and the only one this TCK implements. The schema + * also allows {@code in-process}, a narrow allowance for providers with no backend at all; a + * report claiming it for a provider that has one should be treated with suspicion. + * + * @return the control API kind, always {@code http} + */ + public String controlApi() { + return "http"; + } + /** * Starts the backend with a named configuration, seeding flag state to that configuration's * baseline. diff --git a/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/Outcome.java b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/Outcome.java new file mode 100644 index 000000000..1fe055851 --- /dev/null +++ b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/Outcome.java @@ -0,0 +1,55 @@ +package dev.openfeature.contrib.tools.providertck; + +import com.fasterxml.jackson.annotation.JsonValue; + +/** + * The result of one scenario, or of one capability, in a conformance report. + * + *

There are four rather than two because "did not run" is not one thing. A capability the + * provider chose not to declare is a different statement from one the language makes impossible — + * {@code @strict-numeric-typing} cannot hold in a language with no integer type — and reporting both + * as "not declared" would show a whole language as missing something none of its providers can have. + * + *

The wire values are fixed by the report schema in the OpenFeature specification and are + * identical in every language's TCK. + */ +public enum Outcome { + + /** The scenario ran and every step passed. */ + PASSED("passed"), + + /** The scenario ran and a step failed. */ + FAILED("failed"), + + /** + * The scenario was not run because the provider did not declare the capability it needs. + * + *

Always carries a reason. A skipped scenario reported without one tells a reader that + * something was not checked but not what, which is barely better than omitting it. + */ + NOT_DECLARED("not-declared"), + + /** + * The scenario cannot apply to this provider, because the language makes it unsatisfiable. + * + *

Distinct from {@link #NOT_DECLARED} on purpose: nothing the provider author can do would + * change it, so a comparison page should not display it as a gap. + */ + NOT_APPLICABLE("not-applicable"); + + private final String value; + + Outcome(String value) { + this.value = value; + } + + /** + * Returns the value this outcome is written as in a report. + * + * @return the wire value defined by the report schema + */ + @JsonValue + public String value() { + return value; + } +} diff --git a/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ProviderTckHarness.java b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ProviderTckHarness.java index 1cc671faf..0c42fea71 100644 --- a/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ProviderTckHarness.java +++ b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ProviderTckHarness.java @@ -125,6 +125,23 @@ default Set capabilities() { return EnumSet.allOf(Capability.class); } + /** + * Returns the name of the provider configuration this suite exercises. + * + *

Used to name the conformance report — {@code

/.json} — and reported in + * it as the provider's configuration rather than its identity. The identity is what the provider + * says through its own metadata; this is which of its modes was tested, and a provider with two + * materially different modes produces two reports that are not interchangeable. + * + *

Derived from the suite class name by default: {@code FlagdInProcessTckTest} becomes + * {@code flagd-in-process}. Override it when that does not read well. + * + * @return a short name for this configuration + */ + default String configuration() { + return ReportNames.configurationOf(getClass()); + } + /** * Returns the Compose service name that hosts the control API and the backend the provider * connects to. diff --git a/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ReportNames.java b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ReportNames.java new file mode 100644 index 000000000..7f7e309a1 --- /dev/null +++ b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ReportNames.java @@ -0,0 +1,83 @@ +package dev.openfeature.contrib.tools.providertck; + +import java.util.Locale; + +/** + * Derives the names a conformance report is identified and filed under. + * + *

Kept apart from the report itself so that both the default + * {@link ProviderTckHarness#configuration()} and the file the plugin writes agree on one derivation. + */ +final class ReportNames { + + /** Suffixes a suite class name carries for JUnit's benefit rather than the report's. */ + private static final String[] SUITE_SUFFIXES = {"TckTest", "TCKTest", "TckSuite", "Test", "IT"}; + + /** Used when a name sanitises away to nothing, which an anonymous class manages. */ + private static final String FALLBACK = "provider-tck"; + + private ReportNames() {} + + /** + * Derives a configuration name from a suite class. + * + *

{@code FlagdInProcessTckTest} becomes {@code flagd-in-process}: the suffix that exists only + * so JUnit picks the class up is dropped, and the rest is hyphenated. A provider whose modes do + * not read well this way overrides {@link ProviderTckHarness#configuration()} and says so + * directly. + * + * @param suite the concrete suite class + * @return a hyphenated, lower-case configuration name + */ + static String configurationOf(Class suite) { + String simple = suite.getSimpleName(); + for (String suffix : SUITE_SUFFIXES) { + if (simple.length() > suffix.length() && simple.endsWith(suffix)) { + simple = simple.substring(0, simple.length() - suffix.length()); + break; + } + } + String hyphenated = simple.replaceAll("([a-z0-9])([A-Z])", "$1-$2") + .replaceAll("([A-Z]+)([A-Z][a-z])", "$1-$2") + .toLowerCase(Locale.ROOT); + return hyphenated.isEmpty() ? FALLBACK : hyphenated; + } + + /** + * Turns a configuration name into the file the report is written to. + * + *

Configuration names are chosen to read well in a failure message rather than to be + * path-safe, so anything that is not obviously safe becomes a hyphen. Without this a + * configuration named {@code flagd/rpc} would silently write outside the directory it was given. + * + * @param configuration the configuration name + * @return a file name ending in {@code .json} + */ + static String fileNameOf(String configuration) { + StringBuilder safe = new StringBuilder(configuration.length()); + for (int i = 0; i < configuration.length(); i++) { + char c = configuration.charAt(i); + boolean allowed = (c >= 'a' && c <= 'z') + || (c >= 'A' && c <= 'Z') + || (c >= '0' && c <= '9') + || c == '-' + || c == '_' + || c == '.'; + safe.append(allowed ? c : '-'); + } + String trimmed = trim(safe.toString()); + return (trimmed.isEmpty() ? FALLBACK : trimmed) + ".json"; + } + + private static String trim(String value) { + int start = 0; + int end = value.length(); + while (start < end && (value.charAt(start) == '-' || value.charAt(start) == '.')) { + start++; + } + while (end > start && (value.charAt(end - 1) == '-' || value.charAt(end - 1) == '.')) { + end--; + } + return value.substring(start, end); + } +} diff --git a/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/TckBuildInfo.java b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/TckBuildInfo.java new file mode 100644 index 000000000..bb5e18a0f --- /dev/null +++ b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/TckBuildInfo.java @@ -0,0 +1,135 @@ +package dev.openfeature.contrib.tools.providertck; + +import dev.openfeature.sdk.OpenFeatureAPI; +import java.io.IOException; +import java.io.InputStream; +import java.util.Properties; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * What this TCK build is, and which conformance artifacts it carries. + * + *

A conformance report has to say which questions were asked, not only what the answers were, and + * neither fact is available from the running code. The spec revision the packaged Gherkin came from + * is a property of the build — the artifacts are in the JAR, the repository they came from is not — + * so it is baked into a generated resource at build time and read back here. The SDK version is the + * opposite case: the TCK depends on a version range, so what a consumer actually ran + * against is only knowable at runtime. + * + *

Everything here degrades to {@code unknown} rather than throwing. A report that is slightly + * less identifiable is worth more than a test run that fails while tidying up after itself. + */ +final class TckBuildInfo { + + /** Which TCK implementation this is, in the form the report schema asks for. */ + static final String IMPLEMENTATION = "java-sdk-contrib/tools/provider-tck"; + + /** Maven coordinates of the SDK, reported without a version. */ + static final String SDK_NAME = "dev.openfeature:sdk"; + + /** Stands in for anything the build or the classpath could not tell us. */ + static final String UNKNOWN = "unknown"; + + private static final Logger log = LoggerFactory.getLogger(TckBuildInfo.class); + + /** Generated by the build from the module POM; see {@code src/main/resources-filtered}. */ + private static final String BUILD_PROPERTIES = "provider-tck-build.properties"; + + /** Written into every JAR by maven-archiver, and the most direct statement of what resolved. */ + private static final String SDK_POM_PROPERTIES = "META-INF/maven/dev.openfeature/sdk/pom.properties"; + + private static final Properties BUILD = loadBuildProperties(); + + private TckBuildInfo() {} + + /** Returns the version of this TCK module. */ + static String tckVersion() { + return property("tck.version"); + } + + /** Returns the open-feature/spec commit the packaged conformance artifacts came from. */ + static String specRevision() { + return property("spec.revision"); + } + + /** + * Returns the git tree ID of the conformance artifacts, or {@code null} when it is not recorded. + * + *

Null rather than {@code unknown} because the schema constrains this field to forty hex + * characters, so an unknown value has to be omitted rather than described. + */ + static String assetsTree() { + String tree = BUILD.getProperty("spec.assetsTree", "").trim(); + return tree.matches("[0-9a-f]{40}") ? tree : null; + } + + /** + * Returns the OpenFeature SDK version that was actually on the classpath. + * + *

Read rather than declared. The TCK depends on an SDK version range so that adopting it can + * never force an upgrade, which means the version is a property of the consumer's build; a + * hardcoded one would be a second place to be wrong. + */ + static String sdkVersion() { + Properties pom = load(SDK_POM_PROPERTIES); + if (pom != null) { + String version = pom.getProperty("version", "").trim(); + if (!version.isEmpty()) { + return version; + } + } + + // The Maven descriptor can be stripped, or the SDK can arrive from somewhere that is not a + // JAR at all — an IDE's output directory, say — so fall back to the manifest. + Package sdkPackage = OpenFeatureAPI.class.getPackage(); + String implementation = sdkPackage == null ? null : sdkPackage.getImplementationVersion(); + if (implementation != null && !implementation.trim().isEmpty()) { + return implementation.trim(); + } + + log.warn( + "Could not determine the OpenFeature SDK version from {} or from the package manifest; " + + "the conformance report will say '{}'", + SDK_POM_PROPERTIES, + UNKNOWN); + return UNKNOWN; + } + + private static String property(String key) { + String value = BUILD.getProperty(key, "").trim(); + return value.isEmpty() ? UNKNOWN : value; + } + + private static Properties loadBuildProperties() { + Properties properties = load(packagePath() + BUILD_PROPERTIES); + if (properties == null) { + log.warn( + "{} is missing from the TCK JAR, so a conformance report cannot identify the build " + + "that produced it. This means the resource filtering configured in the " + + "provider-tck POM did not run.", + BUILD_PROPERTIES); + return new Properties(); + } + return properties; + } + + private static String packagePath() { + return TckBuildInfo.class.getPackage().getName().replace('.', '/') + "/"; + } + + private static Properties load(String resource) { + ClassLoader loader = TckBuildInfo.class.getClassLoader(); + try (InputStream in = loader.getResourceAsStream(resource)) { + if (in == null) { + return null; + } + Properties properties = new Properties(); + properties.load(in); + return properties; + } catch (IOException e) { + log.warn("Could not read {} from the classpath", resource, e); + return null; + } + } +} diff --git a/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/TckRunMetadata.java b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/TckRunMetadata.java new file mode 100644 index 000000000..aa6553cc7 --- /dev/null +++ b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/TckRunMetadata.java @@ -0,0 +1,82 @@ +package dev.openfeature.contrib.tools.providertck; + +import java.util.Collections; +import java.util.EnumSet; +import java.util.Optional; +import java.util.Set; + +/** + * What a conformance report needs to know about the run, gathered as the run proceeds. + * + *

It exists because the two facts a report needs most are known at different times and are gone + * by the time it is written. The declared capabilities and the backend description come from the + * harness when the suite starts; the provider's own metadata name is only observable once a scenario + * has built a provider. Meanwhile Cucumber emits {@code TestRunFinished} — where the file is written + * — after {@code @AfterAll} has already torn the {@link TckRuntime} down. + * + *

So the runtime fills this in as it goes and keeps it after {@link TckRuntime#stop()}, and the + * report plugin reads it at the end. + */ +final class TckRunMetadata { + + private final String configuration; + private final Set capabilities; + private final String backendDescription; + private final String controlApi; + + private volatile String providerName; + + TckRunMetadata(String configuration, Set capabilities, String backendDescription, String controlApi) { + this.configuration = configuration; + this.capabilities = capabilities.isEmpty() + ? Collections.emptySet() + : Collections.unmodifiableSet(EnumSet.copyOf(capabilities)); + this.backendDescription = backendDescription; + this.controlApi = controlApi; + } + + /** Returns the suite name, which is the provider configuration under test. */ + String configuration() { + return configuration; + } + + /** Returns the capabilities the harness declared. */ + Set capabilities() { + return capabilities; + } + + /** Returns a short description of the backend stack, or empty when there is none. */ + Optional backendDescription() { + return Optional.ofNullable(backendDescription); + } + + /** Returns how the backend was driven, or empty when there is no control API. */ + Optional controlApi() { + return Optional.ofNullable(controlApi); + } + + /** + * Records what the provider called itself, as observed from a scenario that built one. + * + * @param name the provider's own metadata name + */ + void recordProviderName(String name) { + if (name != null && !name.trim().isEmpty()) { + providerName = name; + } + } + + /** + * Returns the provider's own metadata name, falling back to the configuration name. + * + *

The fallback covers a run in which no scenario ever built a provider — every scenario + * skipped, or the stack never came up. Reporting the suite name there is more useful than the + * empty string the schema would reject. + * + * @return the name to report as the provider's identity + */ + String providerName() { + String observed = providerName; + return observed == null ? configuration : observed; + } +} diff --git a/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/TckRuntime.java b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/TckRuntime.java index 82da5aa11..30efe639e 100644 --- a/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/TckRuntime.java +++ b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/TckRuntime.java @@ -41,10 +41,19 @@ public final class TckRuntime { private static TckRuntime instance; + /** + * What the most recent suite was, kept after {@link #stop()} for the conformance report. + * + *

Cucumber emits the end-of-run event that writes the report after {@code @AfterAll} + * has stopped the runtime, so the report would otherwise have nothing left to describe. + */ + private static volatile TckRunMetadata lastRunMetadata; + private final ProviderTckHarness harness; private final ComposeContainer compose; private final ControlApiClient controlApi; private final BackendEndpoint endpoint; + private final TckRunMetadata metadata; private TckRuntime(ProviderTckHarness harness, ComposeContainer compose) { this.harness = harness; @@ -53,6 +62,12 @@ private TckRuntime(ProviderTckHarness harness, ComposeContainer compose) { String baseUrl = "http://" + compose.getServiceHost(harness.backendService(), null) + ":" + compose.getServicePort(harness.backendService(), harness.controlPort()); this.controlApi = new ControlApiClient(baseUrl, harness.settleTime()); + this.metadata = new TckRunMetadata( + harness.configuration(), + harness.capabilities(), + "Docker Compose stack " + harness.composeFile().getName() + ", service " + harness.backendService(), + controlApi.controlApi()); + lastRunMetadata = this.metadata; } /** @@ -121,6 +136,27 @@ public BackendEndpoint endpoint() { return endpoint; } + /** + * Records what the provider under test calls itself. + * + *

A conformance report identifies the provider by its own metadata name rather than by the + * suite's, and only a scenario that has built one can say what that is. + * + * @param name the provider's metadata name + */ + public void recordProviderName(String name) { + metadata.recordProviderName(name); + } + + /** + * Returns what was observed about the most recent suite, for the conformance report. + * + * @return the metadata of the last suite to start, or empty if none has + */ + static Optional lastRunMetadata() { + return Optional.ofNullable(lastRunMetadata); + } + private static ComposeContainer startCompose(ProviderTckHarness harness) { File composeFile = harness.composeFile(); if (!composeFile.isFile()) { diff --git a/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/steps/ProviderSteps.java b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/steps/ProviderSteps.java index 8fe239414..a88a3aa9c 100644 --- a/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/steps/ProviderSteps.java +++ b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/steps/ProviderSteps.java @@ -149,11 +149,13 @@ public void createProvider(String flavour) { state.provider = provider; state.domain = domain; state.client = api.getClient(domain); - log.info( - "Registered {} provider {} under domain {}", - flavour, - provider.getMetadata().getName(), - domain); + + // A conformance report identifies the provider by what it calls itself, not by the suite + // name, and this is the only place that knows it. + String providerName = provider.getMetadata().getName(); + runtime().recordProviderName(providerName); + + log.info("Registered {} provider {} under domain {}", flavour, providerName, domain); } /** diff --git a/tools/provider-tck/src/main/resources-filtered/dev/openfeature/contrib/tools/providertck/provider-tck-build.properties b/tools/provider-tck/src/main/resources-filtered/dev/openfeature/contrib/tools/providertck/provider-tck-build.properties new file mode 100644 index 000000000..0366d8a4a --- /dev/null +++ b/tools/provider-tck/src/main/resources-filtered/dev/openfeature/contrib/tools/providertck/provider-tck-build.properties @@ -0,0 +1,8 @@ +# Generated at build time by Maven resource filtering. Do not edit the copy in target/. +# +# Identifies the TCK build and the conformance artifacts it carries, for the +# tck section of a conformance report. The values come from the provider-tck +# POM; see the comment there for how the spec revision is maintained. +tck.version=${project.version} +spec.revision=${provider-tck.spec.revision} +spec.assetsTree=${provider-tck.spec.assets-tree} diff --git a/tools/provider-tck/src/test/java/dev/openfeature/contrib/tools/providertck/ConformanceReportPluginTest.java b/tools/provider-tck/src/test/java/dev/openfeature/contrib/tools/providertck/ConformanceReportPluginTest.java new file mode 100644 index 000000000..3f6954b8a --- /dev/null +++ b/tools/provider-tck/src/test/java/dev/openfeature/contrib/tools/providertck/ConformanceReportPluginTest.java @@ -0,0 +1,324 @@ +package dev.openfeature.contrib.tools.providertck; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.entry; + +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.ObjectMapper; +import io.cucumber.plugin.event.Event; +import io.cucumber.plugin.event.EventHandler; +import io.cucumber.plugin.event.EventPublisher; +import io.cucumber.plugin.event.Location; +import io.cucumber.plugin.event.Result; +import io.cucumber.plugin.event.Status; +import io.cucumber.plugin.event.TestCase; +import io.cucumber.plugin.event.TestCaseFinished; +import io.cucumber.plugin.event.TestRunFinished; +import io.cucumber.plugin.event.TestStep; +import java.io.IOException; +import java.net.URI; +import java.nio.file.Files; +import java.nio.file.Path; +import java.time.Duration; +import java.time.Instant; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.EnumSet; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.Set; +import java.util.UUID; +import java.util.stream.Stream; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; +import org.opentest4j.TestAbortedException; + +/** + * The conformance report exists to make one rule checkable, so these tests check it. + * + *

Appendix F requires that a scenario skipped for an undeclared capability is reported as skipped + * with the reason and never as passed. The Go TCK shipped a version of this emitter that broke the + * rule silently — its runner did not deliver the capability-skip signal to the after-hook, so every + * skipped scenario was recorded twice, once correctly and once as passed. The equivalent hazard in + * Cucumber would be an event arriving more than once per scenario, or a hook's success masking the + * abort, so the totals and the per-scenario outcomes are asserted directly rather than assumed. + */ +class ConformanceReportPluginTest { + + private static final Set DECLARED = EnumSet.of(Capability.OBJECT, Capability.EVENTS); + + private static final ObjectMapper MAPPER = new ObjectMapper(); + + @Test + @DisplayName("every scenario appears exactly once, and a skipped one is never called passed") + void everyScenarioAppearsExactlyOnce(@TempDir Path dir) throws IOException { + JsonNode report = run(dir, defaultScenarios()); + + List scenarios = new ArrayList<>(); + report.get("scenarios").forEach(scenarios::add); + + assertThat(scenarios).hasSize(defaultScenarios().size()); + assertThat(scenarios).extracting(s -> s.get("name").asText()).doesNotHaveDuplicates(); + + Map byOutcome = new HashMap<>(); + for (JsonNode scenario : scenarios) { + byOutcome.merge(scenario.get("outcome").asText(), 1L, Long::sum); + } + assertThat(byOutcome).containsOnly(entry("passed", 2L), entry("failed", 1L), entry("not-declared", 2L)); + assertThat(byOutcome.values().stream().mapToLong(Long::longValue).sum()).isEqualTo(scenarios.size()); + } + + @Test + @DisplayName("a capability skip carries the reason it was skipped") + void aCapabilitySkipCarriesItsReason(@TempDir Path dir) throws IOException { + JsonNode report = run(dir, defaultScenarios()); + + JsonNode stale = scenarioNamed(report, "stale scenario"); + assertThat(stale.get("outcome").asText()).isEqualTo("not-declared"); + assertThat(stale.get("reason").asText()) + .isEqualTo("requires capability @stale, which this provider does not declare"); + assertThat(stale.get("tags")).extracting(JsonNode::asText).contains("@stale"); + } + + @Test + @DisplayName("every capability is reported, and only a declared one that failed reads as failed") + void everyCapabilityIsReported(@TempDir Path dir) throws IOException { + JsonNode capabilities = run(dir, defaultScenarios()).get("capabilities"); + + for (Capability capability : Capability.values()) { + JsonNode result = capabilities.get(capability.tag()); + assertThat(result) + .as( + "capability %s is missing; an absent one cannot be told apart from a forgotten one", + capability.tag()) + .isNotNull(); + + String state = result.get("state").asText(); + if (!DECLARED.contains(capability)) { + assertThat(state).isEqualTo("not-declared"); + } else if (capability == Capability.EVENTS) { + // The failing scenario carries @events, so the capability cannot read as passed. + assertThat(state).isEqualTo("failed"); + } else { + assertThat(state).isEqualTo("passed"); + } + + // The schema requires a reason for anything that did not pass, because a bare tag with + // no explanation is not something a person comparing providers can act on. + if ("passed".equals(state)) { + assertThat(result.has("reason")).isFalse(); + } else { + assertThat(result.get("reason").asText()).isNotEmpty(); + } + } + } + + @Test + @DisplayName("a scenario that did not pass always says why") + void everyNonPassingScenarioSaysWhy(@TempDir Path dir) throws IOException { + for (JsonNode scenario : run(dir, defaultScenarios()).get("scenarios")) { + if ("passed".equals(scenario.get("outcome").asText())) { + assertThat(scenario.has("reason")).isFalse(); + } else { + assertThat(scenario.get("reason").asText()) + .as("scenario %s", scenario.get("name").asText()) + .isNotEmpty(); + } + } + } + + @Test + @DisplayName("the report carries everything the schema requires") + void theReportCarriesWhatTheSchemaRequires(@TempDir Path dir) throws IOException { + JsonNode report = run(dir, defaultScenarios()); + + assertThat(report.get("schemaVersion").asText()).isEqualTo("1"); + assertThat(report.get("provider").get("name").asText()).isEqualTo("My Provider"); + assertThat(report.get("provider").get("language").asText()).isEqualTo("java"); + assertThat(report.get("provider").get("configuration").asText()).isEqualTo("my-provider-rpc"); + assertThat(report.get("sdk").get("name").asText()).isEqualTo("dev.openfeature:sdk"); + assertThat(report.get("sdk").get("version").asText()).isNotEmpty(); + assertThat(report.get("tck").get("implementation").asText()).isEqualTo("java-sdk-contrib/tools/provider-tck"); + assertThat(report.get("tck").get("specRevision").asText()).hasSizeGreaterThanOrEqualTo(7); + assertThat(report.get("tck").get("assetsTree").asText()).matches("[0-9a-f]{40}"); + assertThat(report.get("backend").get("controlApi").asText()).isEqualTo("http"); + } + + @Test + @DisplayName("the SDK version is read from the classpath rather than declared") + void theSdkVersionIsRead() { + assertThat(TckBuildInfo.sdkVersion()) + .as("the OpenFeature SDK is on the test classpath, so its version must be discoverable") + .isNotEqualTo(TckBuildInfo.UNKNOWN) + .matches("\\d+\\.\\d+.*"); + } + + @Test + @DisplayName("nothing is written when no report directory is configured") + void nothingIsWrittenByDefault(@TempDir Path dir) throws IOException { + FakeEventPublisher publisher = new FakeEventPublisher(); + new ConformanceReportPlugin(() -> null, () -> Optional.of(metadata())).setEventPublisher(publisher); + publisher.emit(finished(scenario("evaluation", "a scenario", "@object"), Status.PASSED, null)); + publisher.emit(new TestRunFinished(Instant.now(), new Result(Status.PASSED, Duration.ZERO, null))); + + try (Stream written = Files.list(dir)) { + assertThat(written).isEmpty(); + } + } + + @Test + @DisplayName("the report is named after the configuration, safely") + void theReportIsNamedAfterTheConfiguration() { + assertThat(ReportNames.configurationOf(MyProviderRpcTckTest.class)).isEqualTo("my-provider-rpc"); + assertThat(ReportNames.fileNameOf("flagd-rpc")).isEqualTo("flagd-rpc.json"); + assertThat(ReportNames.fileNameOf("flagd/rpc")) + .as("a configuration name is chosen to read well, not to be path-safe") + .isEqualTo("flagd-rpc.json"); + assertThat(ReportNames.fileNameOf("../escape")).isEqualTo("escape.json"); + } + + /** A suite whose name the default configuration derivation has to cope with. */ + private static final class MyProviderRpcTckTest {} + + private JsonNode run(Path dir, List events) throws IOException { + FakeEventPublisher publisher = new FakeEventPublisher(); + new ConformanceReportPlugin(dir::toString, () -> Optional.of(metadata())).setEventPublisher(publisher); + events.forEach(publisher::emit); + publisher.emit(new TestRunFinished(Instant.now(), new Result(Status.FAILED, Duration.ZERO, null))); + + Path written = dir.resolve("my-provider-rpc.json"); + assertThat(written).exists(); + return MAPPER.readTree(Files.readAllBytes(written)); + } + + private static TckRunMetadata metadata() { + TckRunMetadata metadata = new TckRunMetadata("my-provider-rpc", DECLARED, "a test double", "http"); + metadata.recordProviderName("My Provider"); + return metadata; + } + + /** + * The scenarios every test works from: two that pass, one that fails, two skipped for a + * capability the harness did not declare. + */ + private static List defaultScenarios() { + return Arrays.asList( + finished(scenario("evaluation", "a plain scenario"), Status.PASSED, null), + finished(scenario("evaluation", "an object scenario", "@object"), Status.PASSED, null), + finished( + scenario("events", "an event scenario", "@events"), + Status.FAILED, + new AssertionError("expected PROVIDER_READY")), + finished( + scenario("events", "stale scenario", "@events", "@stale"), + Status.SKIPPED, + new TestAbortedException("Skipped: provider does not declare capability STALE")), + finished( + scenario("lifecycle", "a lifecycle scenario", "@lifecycle"), + Status.SKIPPED, + new TestAbortedException("Skipped: provider does not declare capability LIFECYCLE"))); + } + + private static TestCaseFinished finished(TestCase testCase, Status status, Throwable error) { + return new TestCaseFinished(Instant.now(), testCase, new Result(status, Duration.ofMillis(12), error)); + } + + private static TestCase scenario(String feature, String name, String... tags) { + return new FakeTestCase(URI.create("classpath:features/" + feature + ".feature"), name, Arrays.asList(tags)); + } + + private static JsonNode scenarioNamed(JsonNode report, String name) { + for (JsonNode scenario : report.get("scenarios")) { + if (name.equals(scenario.get("name").asText())) { + return scenario; + } + } + throw new AssertionError("no scenario named " + name + " in the report"); + } + + /** Collects the plugin's handlers so a test can drive them directly. */ + private static final class FakeEventPublisher implements EventPublisher { + + private final Map, List>> handlers = new HashMap<>(); + + @Override + public void registerHandlerFor(Class eventType, EventHandler handler) { + handlers.computeIfAbsent(eventType, key -> new ArrayList<>()).add(handler); + } + + @Override + public void removeHandlerFor(Class eventType, EventHandler handler) { + handlers.getOrDefault(eventType, Collections.emptyList()).remove(handler); + } + + @SuppressWarnings("unchecked") + void emit(T event) { + for (EventHandler handler : handlers.getOrDefault(event.getClass(), Collections.emptyList())) { + ((EventHandler) handler).receive(event); + } + } + } + + /** The parts of a Cucumber test case the report reads, and nothing else. */ + private static final class FakeTestCase implements TestCase { + + private final URI uri; + private final String name; + private final List tags; + + FakeTestCase(URI uri, String name, List tags) { + this.uri = uri; + this.name = name; + this.tags = tags; + } + + @Override + public Integer getLine() { + return 1; + } + + @Override + public Location getLocation() { + return new Location(1, 1); + } + + @Override + public String getKeyword() { + return "Scenario"; + } + + @Override + public String getName() { + return name; + } + + @Override + public String getScenarioDesignation() { + return uri + ":1"; + } + + @Override + public List getTags() { + return tags; + } + + @Override + public List getTestSteps() { + return Collections.emptyList(); + } + + @Override + public URI getUri() { + return uri; + } + + @Override + public UUID getId() { + return UUID.randomUUID(); + } + } +} From d071cf29b0c35db3110e335a54c9fc86fdf7bdd4 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Mon, 24 Aug 2026 21:51:32 +0200 Subject: [PATCH 2/3] feat(provider-tck): identify a scenario entry by its Examples row A report entry was identified by feature and name, and every row of a Scenario Outline shares one name. The type-mismatch matrix in errors.feature is eleven rows, so both flagd reports carried eleven entries that nothing distinguished -- 29 entries under 13 distinct names. If one row had failed and ten passed, the report could not have said which, and a consumer building a map from it keeps whichever row it saw last. Each entry now carries `example`, the row's parameters keyed by their Examples column header, as defined by the report schema. Values are the cell contents verbatim, as strings: Gherkin has no types, so "1" stays the string 1 and coercing it would make the report say something the table did not. It is a field rather than a naming convention because the parameters are the identity, and they come from the feature file rather than from any runner. Mandating a mangled name instead would put a separator, an ordering and an escaping rule into normative text that four languages must reproduce byte for byte, with drift invisible until two reports silently fail to line up. The implementations had already diverged on precisely this point before the field existed: Go emitted the bare scenario name for all eleven rows, Python appended its pytest node id, JavaScript its runner's expanded title. Each is a reasonable display name; none of them is a shared identity. Recovering the row takes some care. TestCaseFinished carries a compiled pickle that no longer knows it came from a table, but TestCase.getLocation() resolves the last of the pickle's AST node ids, which the pickle compiler sets to the Examples TableRow -- a plain scenario's last node is the scenario itself, so a line number tells the two apart. ScenarioExamples parses the feature source Cucumber publishes on TestSourceRead, rather than resolving the feature file a second time: Cucumber has already located and decoded it, and re-resolving classpath:features/errors.feature would give a different answer whenever a consumer supplies features from somewhere else. Parsing uses the Gherkin parser Cucumber already depends on, so the report reads the same document the runner executed; io.cucumber:gherkin and io.cucumber:messages were already on the classpath transitively and are now declared, with versions still managed by cucumber-bom. A row skipped for an undeclared capability carries its example too. Eleven skips sharing a name are exactly as ambiguous as eleven failures sharing one. Gherkin also permits a tag on an individual Examples block, so two rows of one outline can differ in whether the capability gate stops them. Nothing here is keyed by scenario name -- one event in, one entry out, and the row lookup is by URI and line -- so a gated row cannot suppress its siblings, and a test covers that case directly. The Go implementation had that bug: its skip bookkeeping was keyed by name, and gating one row dropped every other row of the outline from the report. Verified against the flagd testbed in both resolver modes: 29 entries each, 11 distinct example objects under "Requesting the wrong type returns the code default", (feature, name, example) unique across all 29 where (feature, name) yields only 13, and both reports still valid against the Draft 2020-12 schema. Signed-off-by: Simon Schrottner --- tools/provider-tck/README.md | 21 ++ tools/provider-tck/pom.xml | 23 ++ .../tools/providertck/ConformanceReport.java | 23 +- .../providertck/ConformanceReportPlugin.java | 37 +++- .../tools/providertck/ScenarioExamples.java | 167 ++++++++++++++ .../ConformanceReportPluginTest.java | 206 +++++++++++++++++- 6 files changed, 469 insertions(+), 8 deletions(-) create mode 100644 tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ScenarioExamples.java diff --git a/tools/provider-tck/README.md b/tools/provider-tck/README.md index 0c05c1904..792affe6e 100644 --- a/tools/provider-tck/README.md +++ b/tools/provider-tck/README.md @@ -335,6 +335,27 @@ Note that `capabilities` summarises the *optional* contract only. Scenarios carr tag are mandatory and roll up into nothing, so a provider can fail one while every capability reads `passed`. Read `scenarios` to decide whether a provider conforms. +### What identifies a scenario + +A scenario entry is identified by `feature`, `name` **and** `example` together. The first two are not +enough: every row of a Scenario Outline shares one name, and the type-mismatch matrix in +`errors.feature` is eleven rows. `example` is the row's parameters keyed by its Examples column +header, verbatim as strings — Gherkin has no types, so `"1"` stays a string. + +```console +$ jq '.scenarios[] | select(.feature == "errors") | .example' reports/flagd-rpc.json +{ + "key": "string-flag", + "requested": "Boolean", + "default": "false" +} +... +``` + +It is absent for a scenario that did not come from an outline, and present for every row that did — +including a row skipped for an undeclared capability, since eleven skips sharing a name are exactly +as ambiguous as eleven failures. + ### What identifies a report `provider.name` is what the provider reports through its own metadata, not the suite name. The suite diff --git a/tools/provider-tck/pom.xml b/tools/provider-tck/pom.xml index 7633ff37b..b73180965 100644 --- a/tools/provider-tck/pom.xml +++ b/tools/provider-tck/pom.xml @@ -132,6 +132,29 @@ cucumber-junit-platform-engine + + + io.cucumber + gherkin + + + + io.cucumber + messages + + + io.cucumber diff --git a/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ConformanceReport.java b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ConformanceReport.java index 11ce4a943..cb13eb179 100644 --- a/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ConformanceReport.java +++ b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ConformanceReport.java @@ -192,6 +192,20 @@ public static final class ScenarioResult { /** The scenario name as written in the feature file. */ public final String name; + /** + * The Examples row this entry came from, keyed by column header, or {@code null} for a + * scenario that did not come from a Scenario Outline. + * + *

Part of the entry's identity rather than decoration. Every row of an outline shares one + * name, so eleven rows of the type-mismatch matrix produce eleven entries with the same + * feature and name; without the parameters a report cannot say which of them failed. + * + *

Values are the cell contents verbatim, as strings. Gherkin has no types, so + * {@code "1"} stays the string {@code 1} — coercing it would make the report say something + * the table did not. + */ + public final Map example; + /** The scenario's Gherkin tags, including any inherited from the feature. */ public final List tags; @@ -205,9 +219,16 @@ public static final class ScenarioResult { public final double durationMs; ScenarioResult( - String feature, String name, List tags, Outcome outcome, String reason, double duration) { + String feature, + String name, + Map example, + List tags, + Outcome outcome, + String reason, + double duration) { this.feature = feature; this.name = name; + this.example = example; this.tags = tags; this.outcome = outcome; this.reason = reason; diff --git a/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ConformanceReportPlugin.java b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ConformanceReportPlugin.java index e3a8125b6..bda96ed0b 100644 --- a/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ConformanceReportPlugin.java +++ b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ConformanceReportPlugin.java @@ -5,11 +5,13 @@ import edu.umd.cs.findbugs.annotations.SuppressFBWarnings; import io.cucumber.plugin.ConcurrentEventListener; import io.cucumber.plugin.event.EventPublisher; +import io.cucumber.plugin.event.Location; import io.cucumber.plugin.event.Result; import io.cucumber.plugin.event.Status; import io.cucumber.plugin.event.TestCase; import io.cucumber.plugin.event.TestCaseFinished; import io.cucumber.plugin.event.TestRunFinished; +import io.cucumber.plugin.event.TestSourceRead; import java.io.IOException; import java.io.UncheckedIOException; import java.nio.charset.StandardCharsets; @@ -50,6 +52,12 @@ * the capability-skip signal did not reach the after-hook and every skipped scenario was recorded * twice. * + *

What identifies an entry. Feature and name are not enough: every row of a + * Scenario Outline shares one name, and the type-mismatch matrix in {@code errors.feature} is eleven + * rows. Each entry therefore also carries the Examples row it came from, resolved by + * {@link ScenarioExamples}, so a consumer can tell which row failed rather than keeping whichever it + * saw last. + * * @see open-feature/spec#424 */ public final class ConformanceReportPlugin implements ConcurrentEventListener { @@ -72,6 +80,7 @@ public final class ConformanceReportPlugin implements ConcurrentEventListener { private static final String LANGUAGE = "java"; private final List records = Collections.synchronizedList(new ArrayList<>()); + private final ScenarioExamples examples = new ScenarioExamples(); private final Supplier reportDir; private final Supplier> metadata; @@ -87,6 +96,10 @@ public ConformanceReportPlugin() { @Override public void setEventPublisher(EventPublisher publisher) { + // Cucumber publishes the feature's own text before any scenario in it runs, which is where + // the Examples tables come from. Reading it here rather than resolving the feature file + // again keeps the report reading exactly the source the runner executed. + publisher.registerHandlerFor(TestSourceRead.class, read -> examples.read(read.getUri(), read.getSource())); publisher.registerHandlerFor(TestCaseFinished.class, this::onTestCaseFinished); // The end-of-run event carries the run's own result, which the report has no use for: what // matters is the outcome of each scenario, which TestCaseFinished has already delivered. @@ -116,6 +129,7 @@ private void onTestCaseFinished(TestCaseFinished event) { records.add(new ScenarioRecord( featureName(testCase), testCase.getName(), + exampleOf(testCase), Collections.unmodifiableList(new ArrayList<>(testCase.getTags())), result.getStatus(), messageOf(result), @@ -204,6 +218,7 @@ private static ConformanceReport.ScenarioResult resolve( return new ConformanceReport.ScenarioResult( record.feature, record.name, + record.example, record.tags.isEmpty() ? null : record.tags, outcome, reason, @@ -319,6 +334,18 @@ private static String messageOf(Result result) { return message == null || message.trim().isEmpty() ? error.toString() : message.trim(); } + /** + * Returns the Examples row a test case came from, or {@code null} when it came from none. + * + *

Applies to every outcome, not only to failures. A row skipped for an undeclared capability + * is exactly as ambiguous as one that failed: eleven skips sharing a name say nothing about + * which eleven. + */ + private Map exampleOf(TestCase testCase) { + Location location = testCase.getLocation(); + return location == null ? null : examples.rowAt(testCase.getUri(), location.getLine()); + } + /** Turns {@code classpath:features/errors.feature} into {@code errors}. */ private static String featureName(TestCase testCase) { String uri = testCase.getUri().toString(); @@ -332,15 +359,23 @@ private static String featureName(TestCase testCase) { private static final class ScenarioRecord { private final String feature; private final String name; + private final Map example; private final List tags; private final Status status; private final String message; private final double durationMs; ScenarioRecord( - String feature, String name, List tags, Status status, String message, double durationMs) { + String feature, + String name, + Map example, + List tags, + Status status, + String message, + double durationMs) { this.feature = feature; this.name = name; + this.example = example; this.tags = tags; this.status = status; this.message = message; diff --git a/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ScenarioExamples.java b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ScenarioExamples.java new file mode 100644 index 000000000..2b5319a3c --- /dev/null +++ b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ScenarioExamples.java @@ -0,0 +1,167 @@ +package dev.openfeature.contrib.tools.providertck; + +import io.cucumber.gherkin.GherkinParser; +import io.cucumber.messages.types.Envelope; +import io.cucumber.messages.types.Examples; +import io.cucumber.messages.types.Feature; +import io.cucumber.messages.types.FeatureChild; +import io.cucumber.messages.types.GherkinDocument; +import io.cucumber.messages.types.Scenario; +import io.cucumber.messages.types.TableCell; +import io.cucumber.messages.types.TableRow; +import java.net.URI; +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.Collections; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; +import java.util.stream.Stream; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * Resolves which Examples row a scenario came from, so the report can tell one row from another. + * + *

Every row of a Scenario Outline shares one scenario name. The type-mismatch matrix in + * {@code errors.feature} is eleven rows, so a report identified by feature and name alone carries + * eleven entries that nothing distinguishes: if one row fails and ten pass, the report cannot say + * which failed, and a consumer keying on feature and name keeps whichever it saw last. The + * parameters are the identity, so the report records them. + * + *

How the row is found. Cucumber's {@code TestCaseFinished} carries a + * {@link io.cucumber.plugin.event.TestCase}, and a test case is a compiled pickle that no longer + * knows it came from a table. What it does carry is a URI and a location, and for an outline-derived + * pickle that location is the Examples row: {@code TestCase.getLocation()} resolves the last + * of the pickle's AST node ids, which the pickle compiler sets to the {@code TableRow}. A plain + * scenario's last AST node is the scenario itself. So a line number in a feature file is enough to + * tell the two apart, provided the feature file is parsed as well as executed. + * + *

The source comes from Cucumber's own {@code TestSourceRead} event rather than from a file or a + * classpath resource. Cucumber has already located and decoded the feature — resolving + * {@code classpath:features/errors.feature} a second time would mean reimplementing that lookup, + * with a different answer whenever a consumer supplies features from somewhere else. The event is + * published before any scenario in that feature runs, so the table is always in place by the time a + * result needs it. + * + *

Parsing uses the Gherkin parser Cucumber already depends on, so the report reads the same + * document the runner executed rather than a second interpretation of the syntax. + */ +final class ScenarioExamples { + + private static final Logger log = LoggerFactory.getLogger(ScenarioExamples.class); + + /** + * Examples rows by feature URI, then by the line the row sits on. + * + *

Concurrent because {@link io.cucumber.plugin.ConcurrentEventListener} permits events from + * several threads. The TCK suite pins Cucumber to serial execution, but this class is not the + * place to depend on that. + */ + private final Map>> rowsByUri = new ConcurrentHashMap<>(); + + /** + * Parses a feature file and remembers every Examples row in it. + * + * @param uri the feature's URI, as Cucumber reports it + * @param source the feature file's text + */ + void read(URI uri, String source) { + if (uri == null || source == null) { + return; + } + + Map> rows = new LinkedHashMap<>(); + try (Stream envelopes = GherkinParser.builder() + .includeSource(false) + .includeGherkinDocument(true) + .includePickles(false) + .build() + .parse(uri.toString(), source.getBytes(StandardCharsets.UTF_8))) { + envelopes.forEach(envelope -> envelope.getGherkinDocument() + .flatMap(GherkinDocument::getFeature) + .ifPresent(feature -> collect(feature, rows))); + } catch (RuntimeException e) { + // Cucumber parsed this same source to produce the scenarios, so failing here means the + // two parsers disagree rather than that the feature is broken. Report the entries + // without their parameters rather than failing a run over the report's own metadata: + // the outcomes are still correct, and the warning says why the entries are ambiguous. + log.warn( + "provider-tck: could not parse {} for its Examples tables, so scenarios from that " + + "feature will be reported without the row they came from", + uri, + e); + return; + } + + rowsByUri.put(uri, Collections.unmodifiableMap(rows)); + } + + /** + * Returns the Examples row a scenario came from. + * + * @param uri the feature's URI, as Cucumber reports it + * @param line the line the test case reported as its location + * @return the row's parameters keyed by column header, or {@code null} for a scenario that did + * not come from a Scenario Outline + */ + Map rowAt(URI uri, Integer line) { + if (uri == null || line == null) { + return null; + } + return rowsByUri.getOrDefault(uri, Collections.emptyMap()).get(line); + } + + private static void collect(Feature feature, Map> rows) { + for (FeatureChild child : feature.getChildren()) { + child.getScenario().ifPresent(scenario -> collect(scenario, rows)); + child.getRule().ifPresent(rule -> rule.getChildren() + .forEach(ruleChild -> ruleChild.getScenario().ifPresent(scenario -> collect(scenario, rows)))); + } + } + + private static void collect(Scenario scenario, Map> rows) { + for (Examples examples : scenario.getExamples()) { + List headers = + examples.getTableHeader().map(ScenarioExamples::valuesOf).orElse(Collections.emptyList()); + if (headers.isEmpty()) { + continue; + } + for (TableRow row : examples.getTableBody()) { + Map parameters = parametersOf(headers, valuesOf(row)); + if (!parameters.isEmpty()) { + rows.put(row.getLocation().getLine().intValue(), Collections.unmodifiableMap(parameters)); + } + } + } + } + + /** + * Pairs a row's cells with the column headers, verbatim. + * + *

No coercion and no trimming beyond the parser's own: Gherkin has no types, so {@code "1"} + * is the string {@code 1} and turning it into a number would make the report say something the + * table did not. Keys are in column order, which is how the table reads. + * + *

A row with a different number of cells than the table has headers cannot occur — Gherkin + * rejects such a table before Cucumber compiles a pickle from it — but pairing only as far as + * the shorter of the two keeps a malformed document from throwing out of a reporting path. + */ + private static Map parametersOf(List headers, List cells) { + Map parameters = new LinkedHashMap<>(); + int paired = Math.min(headers.size(), cells.size()); + for (int i = 0; i < paired; i++) { + parameters.put(headers.get(i), cells.get(i)); + } + return parameters; + } + + private static List valuesOf(TableRow row) { + List values = new ArrayList<>(row.getCells().size()); + for (TableCell cell : row.getCells()) { + values.add(cell.getValue()); + } + return values; + } +} diff --git a/tools/provider-tck/src/test/java/dev/openfeature/contrib/tools/providertck/ConformanceReportPluginTest.java b/tools/provider-tck/src/test/java/dev/openfeature/contrib/tools/providertck/ConformanceReportPluginTest.java index 3f6954b8a..0f8c25f73 100644 --- a/tools/provider-tck/src/test/java/dev/openfeature/contrib/tools/providertck/ConformanceReportPluginTest.java +++ b/tools/provider-tck/src/test/java/dev/openfeature/contrib/tools/providertck/ConformanceReportPluginTest.java @@ -14,6 +14,7 @@ import io.cucumber.plugin.event.TestCase; import io.cucumber.plugin.event.TestCaseFinished; import io.cucumber.plugin.event.TestRunFinished; +import io.cucumber.plugin.event.TestSourceRead; import io.cucumber.plugin.event.TestStep; import java.io.IOException; import java.net.URI; @@ -26,6 +27,8 @@ import java.util.Collections; import java.util.EnumSet; import java.util.HashMap; +import java.util.HashSet; +import java.util.LinkedHashMap; import java.util.List; import java.util.Map; import java.util.Optional; @@ -53,6 +56,52 @@ class ConformanceReportPluginTest { private static final ObjectMapper MAPPER = new ObjectMapper(); + private static final URI IDENTITY_URI = URI.create("classpath:features/identity.feature"); + + /** + * A feature whose shape is the one the report has to cope with: an outline whose rows share a + * name across two Examples tables, a plain scenario alongside it, and an outline gated on a + * capability so that a skipped row can be checked as well as a passing one. + * + *

Line numbers are looked up from this text rather than written down, so editing it cannot + * silently make a test assert about the wrong row. + */ + private static final String IDENTITY_FEATURE = String.join( + "\n", + "Feature: Report identity", + "", + " Scenario: An unknown flag key returns the code default", + " Given nothing in particular", + "", + " Scenario Outline: Requesting the wrong type returns the code default", + " Given a -flag with key \"\" and a default value \"\"", + "", + " Examples: a string flag requested as something else", + " | key | requested | default |", + " | string-flag | Boolean | false |", + " | string-flag | Integer | 1 |", + "", + " Examples: a boolean flag requested as something else", + " | key | requested | default |", + " | boolean-flag | String | fallback |", + "", + " Scenario Outline: A gated outline", + " Given nothing in particular", + "", + " @stale", + " Examples: gated by a tag on this block alone", + " | mode |", + " | one |", + "", + " Examples: not gated", + " | mode |", + " | two |", + ""); + + private static final String OUTLINE = "Requesting the wrong type returns the code default"; + + private static final String GATED_OUTLINE = "A gated outline"; + @Test @DisplayName("every scenario appears exactly once, and a skipped one is never called passed") void everyScenarioAppearsExactlyOnce(@TempDir Path dir) throws IOException { @@ -181,10 +230,138 @@ void theReportIsNamedAfterTheConfiguration() { assertThat(ReportNames.fileNameOf("../escape")).isEqualTo("escape.json"); } + @Test + @DisplayName("each row of a Scenario Outline carries the Examples row it came from") + void outlineRowsCarryTheirExample(@TempDir Path dir) throws IOException { + JsonNode report = run(dir, identityScenarios()); + + List rows = scenariosNamed(report, OUTLINE); + assertThat(rows) + .as("three rows ran, so three entries are expected, all under the one outline name") + .hasSize(3); + + assertThat(rows) + .extracting(row -> MAPPER.convertValue(row.get("example"), Map.class)) + .containsExactly( + exampleOf("key", "string-flag", "requested", "Boolean", "default", "false"), + exampleOf("key", "string-flag", "requested", "Integer", "default", "1"), + exampleOf("key", "boolean-flag", "requested", "String", "default", "fallback")); + } + + @Test + @DisplayName("cell contents are reported verbatim as strings, because Gherkin has no types") + void cellsAreNotCoerced(@TempDir Path dir) throws IOException { + JsonNode example = + scenariosNamed(run(dir, identityScenarios()), OUTLINE).get(1).get("example"); + + assertThat(example.get("default").isTextual()) + .as("\"1\" is what the table said; a report that turns it into a number says something else") + .isTrue(); + assertThat(example.get("default").asText()).isEqualTo("1"); + } + + @Test + @DisplayName("a scenario that is not an outline row carries no example") + void aPlainScenarioCarriesNoExample(@TempDir Path dir) throws IOException { + JsonNode plain = scenarioNamed(run(dir, identityScenarios()), "An unknown flag key returns the code default"); + + assertThat(plain.has("example")) + .as("the field is omitted rather than emitted empty; the schema requires at least one property") + .isFalse(); + } + + @Test + @DisplayName("a row skipped for an undeclared capability still says which row it was") + void aSkippedOutlineRowCarriesItsExample(@TempDir Path dir) throws IOException { + List rows = scenariosNamed(run(dir, identityScenarios()), GATED_OUTLINE); + + // Gherkin allows a tag on an individual Examples block, so two rows of one outline can + // differ in whether the capability gate stops them. Both rows must appear, and each must + // say which row it was: a skip that took its sibling with it would be invisible in the + // totals, and a skip that cannot name its row is as ambiguous as a failure that cannot. + assertThat(rows).hasSize(2); + assertThat(rows).extracting(row -> row.get("outcome").asText()).containsExactly("not-declared", "passed"); + assertThat(rows) + .extracting(row -> MAPPER.convertValue(row.get("example"), Map.class)) + .containsExactly(exampleOf("mode", "one"), exampleOf("mode", "two")); + } + + @Test + @DisplayName("feature, name and example together identify a scenario uniquely") + void scenariosAreUniquelyIdentified(@TempDir Path dir) throws IOException { + JsonNode report = run(dir, identityScenarios()); + + // Compared as tuples rather than as joined strings, so no separator can make two + // distinct entries look alike or one entry look like two. + List> identities = new ArrayList<>(); + List> namesOnly = new ArrayList<>(); + for (JsonNode scenario : report.get("scenarios")) { + String feature = scenario.get("feature").asText(); + String name = scenario.get("name").asText(); + String example = scenario.has("example") ? scenario.get("example").toString() : ""; + namesOnly.add(Arrays.asList(feature, name)); + identities.add(Arrays.asList(feature, name, example)); + } + + assertThat(new HashSet<>(namesOnly)) + .as("the premise of this test: feature and name alone are not unique in this report") + .hasSizeLessThan(namesOnly.size()); + assertThat(identities) + .as("a consumer keying on feature, name and example must not lose an entry") + .doesNotHaveDuplicates(); + } + /** A suite whose name the default configuration derivation has to cope with. */ private static final class MyProviderRpcTckTest {} - private JsonNode run(Path dir, List events) throws IOException { + /** + * The events the identity tests work from: the feature source Cucumber would publish, then one + * test case per compiled pickle, each located on the line it came from. + */ + private static List identityScenarios() { + return Arrays.asList( + new TestSourceRead(Instant.now(), IDENTITY_URI, IDENTITY_FEATURE), + finished( + identity("An unknown flag key returns the code default", " Scenario: An unknown flag"), + Status.PASSED, + null), + finished(identity(OUTLINE, "| string-flag | Boolean"), Status.PASSED, null), + finished( + identity(OUTLINE, "| string-flag | Integer"), + Status.FAILED, + new AssertionError("resolved 1 with no error code")), + finished(identity(OUTLINE, "| boolean-flag | String"), Status.PASSED, null), + finished( + identity(GATED_OUTLINE, "| one |", "@stale"), + Status.SKIPPED, + new TestAbortedException("Skipped: provider does not declare capability STALE")), + finished(identity(GATED_OUTLINE, "| two |"), Status.PASSED, null)); + } + + private static TestCase identity(String name, String marker, String... tags) { + return new FakeTestCase(IDENTITY_URI, name, Arrays.asList(tags), lineOf(marker)); + } + + /** Finds the 1-based line the given text sits on, so no test hard-codes a line number. */ + private static int lineOf(String marker) { + String[] lines = IDENTITY_FEATURE.split("\n", -1); + for (int i = 0; i < lines.length; i++) { + if (lines[i].contains(marker)) { + return i + 1; + } + } + throw new AssertionError("no line of the test feature contains " + marker); + } + + private static Map exampleOf(String... keysAndValues) { + Map example = new LinkedHashMap<>(); + for (int i = 0; i < keysAndValues.length; i += 2) { + example.put(keysAndValues[i], keysAndValues[i + 1]); + } + return example; + } + + private JsonNode run(Path dir, List events) throws IOException { FakeEventPublisher publisher = new FakeEventPublisher(); new ConformanceReportPlugin(dir::toString, () -> Optional.of(metadata())).setEventPublisher(publisher); events.forEach(publisher::emit); @@ -228,7 +405,7 @@ private static TestCaseFinished finished(TestCase testCase, Status status, Throw } private static TestCase scenario(String feature, String name, String... tags) { - return new FakeTestCase(URI.create("classpath:features/" + feature + ".feature"), name, Arrays.asList(tags)); + return new FakeTestCase(URI.create("classpath:features/" + feature + ".feature"), name, Arrays.asList(tags), 1); } private static JsonNode scenarioNamed(JsonNode report, String name) { @@ -240,6 +417,16 @@ private static JsonNode scenarioNamed(JsonNode report, String name) { throw new AssertionError("no scenario named " + name + " in the report"); } + private static List scenariosNamed(JsonNode report, String name) { + List matching = new ArrayList<>(); + for (JsonNode scenario : report.get("scenarios")) { + if (name.equals(scenario.get("name").asText())) { + matching.add(scenario); + } + } + return matching; + } + /** Collects the plugin's handlers so a test can drive them directly. */ private static final class FakeEventPublisher implements EventPublisher { @@ -269,21 +456,28 @@ private static final class FakeTestCase implements TestCase { private final URI uri; private final String name; private final List tags; + private final int line; - FakeTestCase(URI uri, String name, List tags) { + FakeTestCase(URI uri, String name, List tags, int line) { this.uri = uri; this.name = name; this.tags = tags; + this.line = line; } @Override public Integer getLine() { - return 1; + return line; } + /** + * The pickle's location, which for an outline-derived test case is the Examples row rather + * than the {@code Scenario Outline} line. That is what Cucumber reports, and it is the only + * thing tying a compiled pickle back to the table it came from. + */ @Override public Location getLocation() { - return new Location(1, 1); + return new Location(line, 1); } @Override @@ -298,7 +492,7 @@ public String getName() { @Override public String getScenarioDesignation() { - return uri + ":1"; + return uri + ":" + line; } @Override From d7b5ae2f7434c3cd2cfc6f5bdb4b28e933833494 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Mon, 24 Aug 2026 21:54:59 +0200 Subject: [PATCH 3/3] fix(provider-tck): stop reporting a capability nothing exercised as passed A declared capability read `passed` whether or not anything had examined it. That is the vacuous pass the capability vocabulary was introduced to eliminate, arriving through the report rather than through the suite, and it reached that state two ways. @targeting is reserved: it exists in the tag vocabulary but no scenario carries it, because asserting that an evaluation context reached the backend needs an echo operation the control API does not have. A provider declaring it got a green result for free. @caching is the same. The second route is subtler. A scenario can carry two capability tags and be skipped for the one the provider did not declare, and counting a capability as exercised because a scenario *carried* its tag counts that skip. events.feature is exactly this shape -- the feature is tagged @events and each of its two scenarios adds @stale or @configuration-change -- so a provider declaring @events alone ran neither scenario and was told @events passed. Exercising is now counted by execution: a scenario contributes to its capabilities only when its outcome is passed or failed. Such a capability is omitted from `capabilities` entirely. Nothing asked the question, so there is no answer to report, and a consumer sees the tag is absent rather than a pass it cannot rely on. Omitting is preferred to inventing a fifth outcome: the four in the schema describe what the provider did, and "nothing asked this of the provider" is a fact about the run. Everything else about the rollup is unchanged. Undeclared is still not-declared with a reason; declared, exercised and failing is still failed with a reason, now saying how many of how many ran; declared, exercised and passing is still passed. Follows go-sdk-contrib#944, which made the same two changes there. The second was found by the Python implementation, whose in-memory self-test declares @events without @stale and so hits it directly. flagd declares every capability but @strict-numeric-typing, so its reports now carry seven entries rather than nine. The second route does not change them -- the one undeclared capability is the only tag on its scenario -- so it is latent there, and live for a provider that declares @events without @stale. Signed-off-by: Simon Schrottner --- tools/provider-tck/README.md | 14 ++++ .../providertck/ConformanceReportPlugin.java | 49 ++++++++++++-- .../ConformanceReportPluginTest.java | 66 ++++++++++++++++++- 3 files changed, 120 insertions(+), 9 deletions(-) diff --git a/tools/provider-tck/README.md b/tools/provider-tck/README.md index 792affe6e..6ad6c78a7 100644 --- a/tools/provider-tck/README.md +++ b/tools/provider-tck/README.md @@ -335,6 +335,20 @@ Note that `capabilities` summarises the *optional* contract only. Scenarios carr tag are mandatory and roll up into nothing, so a provider can fail one while every capability reads `passed`. Read `scenarios` to decide whether a provider conforms. +A capability you declare that **no scenario actually exercised** is left out of `capabilities` +entirely rather than reported as `passed`. Reporting a green result for a claim nothing examined is +the vacuous pass the capability vocabulary exists to eliminate, so the suite says nothing instead. +That happens two ways: + +- Nothing in the suite carries the tag. `@targeting` and `@caching` are reserved — they are in the + vocabulary so it stays aligned with the flagd test harness, but no scenario carries them yet. +- Every scenario carrying it was skipped for a *different* capability you did not declare. Both + scenarios in `events.feature` carry `@events` plus one of `@stale` or `@configuration-change`, so + declaring `@events` on its own runs neither, and a run that ran neither has demonstrated nothing + about `@events`. + +Exercising is therefore counted by execution, not by tag presence. + ### What identifies a scenario A scenario entry is identified by `feature`, `name` **and** `example` together. The first two are not diff --git a/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ConformanceReportPlugin.java b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ConformanceReportPlugin.java index bda96ed0b..ddb26765e 100644 --- a/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ConformanceReportPlugin.java +++ b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ConformanceReportPlugin.java @@ -179,8 +179,12 @@ ConformanceReport build(TckRunMetadata run) { // not pass. Map failed = new EnumMap<>(Capability.class); + // And only a capability some scenario carries can be said to have been tested at all. A + // capability nothing in the suite exercises has no outcome to report — see capabilitiesOf. + Map exercised = new EnumMap<>(Capability.class); + for (ScenarioRecord record : observed) { - scenarios.add(resolve(record, declared, failed)); + scenarios.add(resolve(record, declared, exercised, failed)); } return new ConformanceReport( @@ -192,12 +196,15 @@ ConformanceReport build(TckRunMetadata run) { TckBuildInfo.specRevision(), TckBuildInfo.assetsTree()), backendOf(run), - capabilitiesOf(declared, failed), + capabilitiesOf(declared, exercised, failed), Collections.unmodifiableList(scenarios)); } private static ConformanceReport.ScenarioResult resolve( - ScenarioRecord record, Set declared, Map failed) { + ScenarioRecord record, + Set declared, + Map exercised, + Map failed) { Outcome outcome; String reason; @@ -210,8 +217,20 @@ private static ConformanceReport.ScenarioResult resolve( } else { outcome = Outcome.FAILED; reason = record.message == null ? "the scenario was reported as " + record.status : record.message; + } + + // A scenario counts towards its capabilities only if it actually ran. Carrying the tag is + // not the same as exercising the capability: a scenario can carry two, and be skipped for + // the one this provider did not declare. Every scenario in events.feature is like that — + // the feature carries @events and each scenario adds @stale or @configuration-change — so + // counting by tag presence would report @events as passed for a provider that declared it + // and ran neither scenario. + if (outcome == Outcome.PASSED || outcome == Outcome.FAILED) { for (Capability capability : gatingCapabilities(record.tags)) { - failed.merge(capability, 1, Integer::sum); + exercised.merge(capability, 1, Integer::sum); + if (outcome == Outcome.FAILED) { + failed.merge(capability, 1, Integer::sum); + } } } @@ -257,9 +276,21 @@ private static List gatingCapabilities(List tags) { * scenario with no capability tag is mandatory and rolls up into nothing here, so a provider can * fail one while every entry below reads {@code passed}. The per-scenario list is what a * consumer has to read to decide whether a provider conforms. + * + *

A capability nothing exercised is left out. {@code @targeting} is + * reserved: it is in the tag vocabulary but no scenario carries it, because asserting that an + * evaluation context reached the backend needs an echo operation the control API does not have. + * A provider declaring it used to get {@code passed} for a claim nothing had examined — the + * vacuous pass the capability vocabulary exists to eliminate, arriving through the report rather + * than through the suite. The same pass arrives by a second route when every scenario carrying a + * declared capability was skipped for a different capability the provider did not + * declare, which is why exercising is counted by execution and not by tag presence. + * + *

Omitting is preferred to inventing a fifth outcome: the four in the schema describe what + * the provider did, and "nothing asked this of the provider" is a fact about the run. */ private static Map capabilitiesOf( - Set declared, Map failed) { + Set declared, Map exercised, Map failed) { Map results = new LinkedHashMap<>(); for (Capability capability : Capability.values()) { ConformanceReport.CapabilityResult result; @@ -268,11 +299,17 @@ private static Map capabilitiesOf( Outcome.NOT_DECLARED, "not declared by this provider's configuration; the " + capability.tag() + " scenarios were skipped and did not contribute to this result"); + } else if (!exercised.containsKey(capability)) { + // Declared, but no scenario carrying it ran — either nothing in the suite gates on + // it, or everything that does was skipped for some other capability this provider + // did not declare. Either way nothing was demonstrated, so there is nothing to + // report: a consumer sees the tag is absent rather than a pass it cannot rely on. + continue; } else if (failed.containsKey(capability)) { int count = failed.get(capability); result = new ConformanceReport.CapabilityResult( Outcome.FAILED, - count + (count == 1 ? " scenario" : " scenarios") + " carrying " + capability.tag() + count + " of " + exercised.get(capability) + " scenarios carrying " + capability.tag() + " failed; the per-scenario results say which, and why"); } else { result = new ConformanceReport.CapabilityResult(Outcome.PASSED, null); diff --git a/tools/provider-tck/src/test/java/dev/openfeature/contrib/tools/providertck/ConformanceReportPluginTest.java b/tools/provider-tck/src/test/java/dev/openfeature/contrib/tools/providertck/ConformanceReportPluginTest.java index 0f8c25f73..13eb2db41 100644 --- a/tools/provider-tck/src/test/java/dev/openfeature/contrib/tools/providertck/ConformanceReportPluginTest.java +++ b/tools/provider-tck/src/test/java/dev/openfeature/contrib/tools/providertck/ConformanceReportPluginTest.java @@ -142,7 +142,8 @@ void everyCapabilityIsReported(@TempDir Path dir) throws IOException { JsonNode result = capabilities.get(capability.tag()); assertThat(result) .as( - "capability %s is missing; an absent one cannot be told apart from a forgotten one", + "capability %s is missing; a capability is left out only when it is declared and " + + "no scenario exercises it, which is not the case here", capability.tag()) .isNotNull(); @@ -152,6 +153,9 @@ void everyCapabilityIsReported(@TempDir Path dir) throws IOException { } else if (capability == Capability.EVENTS) { // The failing scenario carries @events, so the capability cannot read as passed. assertThat(state).isEqualTo("failed"); + assertThat(result.get("reason").asText()) + .as("someone comparing providers wants to know how much failed before opening the detail") + .contains("1 of 1"); } else { assertThat(state).isEqualTo("passed"); } @@ -311,6 +315,54 @@ void scenariosAreUniquelyIdentified(@TempDir Path dir) throws IOException { .doesNotHaveDuplicates(); } + @Test + @DisplayName("a declared capability no scenario exercises is omitted rather than called passed") + void anUnexercisedCapabilityIsOmitted(@TempDir Path dir) throws IOException { + Set declared = EnumSet.of(Capability.OBJECT, Capability.TARGETING); + JsonNode capabilities = run(dir, declared, defaultScenarios()).get("capabilities"); + + assertThat(capabilities.has(Capability.TARGETING.tag())) + .as( + "%s is declared and no scenario carries it; the suite asked no question, so it has " + + "no answer, and a green result there is a pass nothing examined", + Capability.TARGETING.tag()) + .isFalse(); + + assertThat(capabilities.get(Capability.OBJECT.tag()).get("state").asText()) + .as("the omission must be specific, not a general failure to report capabilities") + .isEqualTo("passed"); + } + + @Test + @DisplayName("a declared capability whose every scenario was skipped is omitted too") + void aCapabilityWhoseScenariosAllSkippedIsOmitted(@TempDir Path dir) throws IOException { + // @events is declared; @stale is not. Every scenario in events.feature carries both — the + // feature is tagged @events and each scenario adds @stale or @configuration-change — so + // declaring @events alone runs none of them. Counting a capability as exercised because a + // scenario carried its tag would report @events as passed here, which is the same vacuous + // pass as a reserved capability arriving by a different route. + Set declared = EnumSet.of(Capability.OBJECT, Capability.EVENTS); + List scenarios = Arrays.asList( + finished(scenario("evaluation", "an object scenario", "@object"), Status.PASSED, null), + finished( + scenario("events", "a stale scenario", "@events", "@stale"), + Status.SKIPPED, + new TestAbortedException("Skipped: provider does not declare capability STALE"))); + + JsonNode capabilities = run(dir, declared, scenarios).get("capabilities"); + + assertThat(capabilities.has(Capability.EVENTS.tag())) + .as( + "%s was declared and no scenario carrying it ran, so nothing was demonstrated", + Capability.EVENTS.tag()) + .isFalse(); + assertThat(capabilities.get(Capability.OBJECT.tag()).get("state").asText()) + .isEqualTo("passed"); + assertThat(capabilities.get(Capability.STALE.tag()).get("state").asText()) + .as("the undeclared capability is still reported, with the reason it was not") + .isEqualTo("not-declared"); + } + /** A suite whose name the default configuration derivation has to cope with. */ private static final class MyProviderRpcTckTest {} @@ -362,8 +414,12 @@ private static Map exampleOf(String... keysAndValues) { } private JsonNode run(Path dir, List events) throws IOException { + return run(dir, DECLARED, events); + } + + private JsonNode run(Path dir, Set declared, List events) throws IOException { FakeEventPublisher publisher = new FakeEventPublisher(); - new ConformanceReportPlugin(dir::toString, () -> Optional.of(metadata())).setEventPublisher(publisher); + new ConformanceReportPlugin(dir::toString, () -> Optional.of(metadata(declared))).setEventPublisher(publisher); events.forEach(publisher::emit); publisher.emit(new TestRunFinished(Instant.now(), new Result(Status.FAILED, Duration.ZERO, null))); @@ -373,7 +429,11 @@ private JsonNode run(Path dir, List events) throws IOException } private static TckRunMetadata metadata() { - TckRunMetadata metadata = new TckRunMetadata("my-provider-rpc", DECLARED, "a test double", "http"); + return metadata(DECLARED); + } + + private static TckRunMetadata metadata(Set declared) { + TckRunMetadata metadata = new TckRunMetadata("my-provider-rpc", declared, "a test double", "http"); metadata.recordProviderName("My Provider"); return metadata; }