diff --git a/tools/provider-tck/README.md b/tools/provider-tck/README.md index e9d8232c3..6ad6c78a7 100644 --- a/tools/provider-tck/README.md +++ b/tools/provider-tck/README.md @@ -295,6 +295,104 @@ documented. The Compose stack starts once per suite and is never restarted. Scenario isolation comes from the control API. +## Conformance reports + +Set `PROVIDER_TCK_REPORT_DIR` and each suite writes a machine-readable report of its run to +`/.json`, conforming to the [report schema][report-schema] in the specification. + +```console +$ PROVIDER_TCK_REPORT_DIR=./reports mvn test -Dtest='Flagd*TckTest' +$ jq '.scenarios | group_by(.outcome) | map({(.[0].outcome): length}) | add' reports/flagd-rpc.json +{ + "passed": 28, + "not-declared": 1 +} +``` + +`-Dprovider.tck.report.dir=...` does the same thing and is often easier to pass through Maven. The +environment variable is the portable spelling — every language's TCK reads it, so one cross-language +CI job can set one thing. + +It is an environment variable rather than a method on `ProviderTckHarness` so that emitting a report +is a property of the run and not of the code: CI sets it, a developer running the suite locally does +not, and no adopter changes a line to publish one. Unset means no report, which is not an error. +Several suites in one JVM each write their own file, so flagd's two resolvers do not collide. + +### What the report is for + +The per-scenario list is the load-bearing part. This suite promises that a scenario skipped for an +undeclared capability is reported as skipped with the reason and *never* as passed — and a promise is +not a check. The report records the outcome of every scenario individually, so a consumer can verify +the rule instead of trusting a runner's headline number. Go's runner counts capability-gated skips in +its **passed** tally, which is exactly the failure mode this makes impossible to hide; Cucumber +reports skips correctly, and the report is what proves it rather than assuming it. + +Every scenario appears exactly once, whatever happened to it. A report that quietly omitted the +scenarios it did not run would satisfy every rule above and still mislead, because a reader would +have no way to know how many questions went unasked. + +Note that `capabilities` summarises the *optional* contract only. Scenarios carrying no capability +tag are mandatory and roll up into nothing, so a provider can fail one while every capability reads +`passed`. Read `scenarios` to decide whether a provider conforms. + +A capability you declare that **no scenario actually exercised** is left out of `capabilities` +entirely rather than reported as `passed`. Reporting a green result for a claim nothing examined is +the vacuous pass the capability vocabulary exists to eliminate, so the suite says nothing instead. +That happens two ways: + +- Nothing in the suite carries the tag. `@targeting` and `@caching` are reserved — they are in the + vocabulary so it stays aligned with the flagd test harness, but no scenario carries them yet. +- Every scenario carrying it was skipped for a *different* capability you did not declare. Both + scenarios in `events.feature` carry `@events` plus one of `@stale` or `@configuration-change`, so + declaring `@events` on its own runs neither, and a run that ran neither has demonstrated nothing + about `@events`. + +Exercising is therefore counted by execution, not by tag presence. + +### What identifies a scenario + +A scenario entry is identified by `feature`, `name` **and** `example` together. The first two are not +enough: every row of a Scenario Outline shares one name, and the type-mismatch matrix in +`errors.feature` is eleven rows. `example` is the row's parameters keyed by its Examples column +header, verbatim as strings — Gherkin has no types, so `"1"` stays a string. + +```console +$ jq '.scenarios[] | select(.feature == "errors") | .example' reports/flagd-rpc.json +{ + "key": "string-flag", + "requested": "Boolean", + "default": "false" +} +... +``` + +It is absent for a scenario that did not come from an outline, and present for every row that did — +including a row skipped for an undeclared capability, since eleven skips sharing a name are exactly +as ambiguous as eleven failures. + +### What identifies a report + +`provider.name` is what the provider reports through its own metadata, not the suite name. The suite +name is chosen to read well in a failure message — `flagd-rpc` — which makes it the *configuration*, +and it is reported as such. One provider with two materially different modes produces two reports +that are not interchangeable. It is derived from the suite class name (`FlagdInProcessTckTest` → +`flagd-in-process`) and can be overridden with `ProviderTckHarness.configuration()`. + +`tck.specRevision` and `tck.assetsTree` identify the conformance artifacts that were executed, and +are baked into the JAR at build time from the properties in this module's POM — the artifacts travel +in the JAR, the repository they came from does not. They are pinned by hand for now because, unlike +the Go TCK, this module has no spec submodule to read them from; the artifacts under +`src/main/resources` are vendored copies. See [Where these artifacts should live](#where-these-artifacts-should-live). +Both are checkable rather than merely asserted: +`git rev-parse :specification/assets/provider-tck` must reproduce the tree, and the +tree must match the files in this module. + +`sdk.version` is read from the classpath rather than declared, because the TCK depends on an SDK +version *range* so that adopting it can never force an upgrade — what a consumer actually ran against +is only knowable at runtime. + +[report-schema]: https://github.com/open-feature/spec/blob/main/specification/assets/provider-tck/report/conformance-report.schema.json + ## Relationship to the flagd test harness The step vocabulary is inherited from the diff --git a/tools/provider-tck/pom.xml b/tools/provider-tck/pom.xml index 3de834808..b73180965 100644 --- a/tools/provider-tck/pom.xml +++ b/tools/provider-tck/pom.xml @@ -14,6 +14,35 @@ ${groupId}.providertck + + + dfa16586d91ca020ef1b3b82a7c972d833ff8f29 + 904aa7d5fd7a856a4f92ace24355bd1987143abc + 3.27.7 4.3.0 2.22.1 @@ -103,6 +132,29 @@ cucumber-junit-platform-engine + + + io.cucumber + gherkin + + + + io.cucumber + messages + + + io.cucumber @@ -182,4 +234,23 @@ + + + + + src/main/resources + false + + + src/main/resources-filtered + true + + + + diff --git a/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/AbstractProviderTckTest.java b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/AbstractProviderTckTest.java index 4f3304d94..bd214c990 100644 --- a/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/AbstractProviderTckTest.java +++ b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/AbstractProviderTckTest.java @@ -33,12 +33,19 @@ * {@code dev.openfeature.contrib.tools.providertck.steps}, which reach the harness through * {@link TckRuntime}. * + *

The suite also carries {@link ConformanceReportPlugin}, so an adopter needs no configuration to + * publish a machine-readable conformance report: setting {@code PROVIDER_TCK_REPORT_DIR} on a run is + * enough, and leaving it unset writes nothing. + * * @see ProviderTckHarness + * @see ConformanceReportPlugin */ @Suite @IncludeEngines("cucumber") @SelectClasspathResource("features") -@ConfigurationParameter(key = Constants.PLUGIN_PROPERTY_NAME, value = "summary") +@ConfigurationParameter( + key = Constants.PLUGIN_PROPERTY_NAME, + value = "summary," + "dev.openfeature.contrib.tools.providertck.ConformanceReportPlugin") @ConfigurationParameter(key = Constants.PARALLEL_EXECUTION_ENABLED_PROPERTY_NAME, value = "false") @ConfigurationParameter(key = Constants.EXECUTION_MODE_FEATURE_PROPERTY_NAME, value = "same_thread") @ConfigurationParameter(key = Constants.GLUE_PROPERTY_NAME, value = "dev.openfeature.contrib.tools.providertck.steps") diff --git a/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ConformanceReport.java b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ConformanceReport.java new file mode 100644 index 000000000..cb13eb179 --- /dev/null +++ b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ConformanceReport.java @@ -0,0 +1,238 @@ +package dev.openfeature.contrib.tools.providertck; + +import com.fasterxml.jackson.annotation.JsonInclude; +import java.util.List; +import java.util.Map; + +/** + * One run of the conformance suite against one provider in one configuration. + * + *

The field names and nesting are fixed by the report schema in the OpenFeature specification + * repository. This class is deliberately a transcription of that schema rather than a shape that + * would be convenient in Java, because the point of the format is that every language's TCK emits + * the same document. + * + *

Fields left {@code null} are omitted from the JSON. The schema sets + * {@code additionalProperties: false} throughout, so an unexpected field is a validation failure + * rather than something a consumer ignores. + * + * @see open-feature/spec#424 + */ +@JsonInclude(JsonInclude.Include.NON_NULL) +public final class ConformanceReport { + + /** + * The major version of the report schema this document conforms to. + * + *

An integer as a string, so a consumer can reject a report it does not understand rather + * than guessing at it. + */ + public static final String SCHEMA_VERSION = "1"; + + /** The schema version this document claims. */ + public final String schemaVersion; + + /** What was tested. */ + public final Provider provider; + + /** Which OpenFeature SDK the provider was exercised through. */ + public final Sdk sdk; + + /** What asked the questions, and which questions. */ + public final Tck tck; + + /** What the provider was pointed at. */ + public final Backend backend; + + /** Per-capability outcome, keyed by Gherkin tag including the leading at-sign. */ + public final Map capabilities; + + /** Per-scenario outcome, one entry per scenario in the suite. */ + public final List scenarios; + + ConformanceReport( + Provider provider, + Sdk sdk, + Tck tck, + Backend backend, + Map capabilities, + List scenarios) { + this.schemaVersion = SCHEMA_VERSION; + this.provider = provider; + this.sdk = sdk; + this.tck = tck; + this.backend = backend; + this.capabilities = capabilities; + this.scenarios = scenarios; + } + + /** Identifies the provider under test. */ + @JsonInclude(JsonInclude.Include.NON_NULL) + public static final class Provider { + + /** + * What the provider calls itself, through its own metadata. + * + *

Not the name of the suite. A suite name is chosen to read well in a failure message — + * {@code flagd-rpc} — which makes it the {@link #configuration}, not the identity. + */ + public final String name; + + /** The language the provider is written in, always {@code java} here. */ + public final String language; + + /** + * Which configuration of the provider was tested. + * + *

A provider with more than one materially different mode produces one report per mode, + * and they are not interchangeable: flagd's RPC and in-process resolvers differ in whether + * they emit {@code PROVIDER_STALE}, so a report keyed on the provider name alone would have + * to pick one and misrepresent the other. + */ + public final String configuration; + + Provider(String name, String language, String configuration) { + this.name = name; + this.language = language; + this.configuration = configuration; + } + } + + /** Identifies the OpenFeature SDK the run went through. */ + @JsonInclude(JsonInclude.Include.NON_NULL) + public static final class Sdk { + + /** Maven coordinates of the SDK, without a version. */ + public final String name; + + /** The resolved SDK version. */ + public final String version; + + Sdk(String name, String version) { + this.name = name; + this.version = version; + } + } + + /** Identifies the TCK implementation and the conformance artifacts it executed. */ + @JsonInclude(JsonInclude.Include.NON_NULL) + public static final class Tck { + + /** Which TCK implementation produced this report. */ + public final String implementation; + + /** The version of that implementation. */ + public final String version; + + /** The open-feature/spec commit the executed artifacts came from. */ + public final String specRevision; + + /** + * The git tree object ID of {@code specification/assets/provider-tck} at that revision. + * + *

Carried alongside the commit because it identifies the artifacts rather than the + * commit: it is unchanged by unrelated edits elsewhere in the specification, and + * {@code git rev-parse :specification/assets/provider-tck} must reproduce it, + * so a revision recorded wrongly does not go unnoticed. + */ + public final String assetsTree; + + Tck(String implementation, String version, String specRevision, String assetsTree) { + this.implementation = implementation; + this.version = version; + this.specRevision = specRevision; + this.assetsTree = assetsTree; + } + } + + /** Describes what the provider was pointed at. */ + @JsonInclude(JsonInclude.Include.NON_NULL) + public static final class Backend { + + /** A short human-readable description of the stack under test. */ + public final String description; + + /** + * How the backend was driven, {@code http} or {@code in-process}. + * + *

{@code in-process} is the narrow allowance made for providers with no backend; a report + * claiming it for a provider that has one should be treated with suspicion. + */ + public final String controlApi; + + Backend(String description, String controlApi) { + this.description = description; + this.controlApi = controlApi; + } + } + + /** The outcome of one capability, with the reason it is not simply {@code passed}. */ + @JsonInclude(JsonInclude.Include.NON_NULL) + public static final class CapabilityResult { + + /** The capability's outcome across every scenario that gates on it. */ + public final Outcome state; + + /** Why, in a form someone reading a comparison page can use. */ + public final String reason; + + CapabilityResult(Outcome state, String reason) { + this.state = state; + this.reason = reason; + } + } + + /** The outcome of one scenario. */ + @JsonInclude(JsonInclude.Include.NON_NULL) + public static final class ScenarioResult { + + /** The feature file the scenario came from, without extension. */ + public final String feature; + + /** The scenario name as written in the feature file. */ + public final String name; + + /** + * The Examples row this entry came from, keyed by column header, or {@code null} for a + * scenario that did not come from a Scenario Outline. + * + *

Part of the entry's identity rather than decoration. Every row of an outline shares one + * name, so eleven rows of the type-mismatch matrix produce eleven entries with the same + * feature and name; without the parameters a report cannot say which of them failed. + * + *

Values are the cell contents verbatim, as strings. Gherkin has no types, so + * {@code "1"} stays the string {@code 1} — coercing it would make the report say something + * the table did not. + */ + public final Map example; + + /** The scenario's Gherkin tags, including any inherited from the feature. */ + public final List tags; + + /** What happened. */ + public final Outcome outcome; + + /** For a skip, why it was skipped; for a failure, what failed. */ + public final String reason; + + /** How long the scenario took. */ + public final double durationMs; + + ScenarioResult( + String feature, + String name, + Map example, + List tags, + Outcome outcome, + String reason, + double duration) { + this.feature = feature; + this.name = name; + this.example = example; + this.tags = tags; + this.outcome = outcome; + this.reason = reason; + this.durationMs = duration; + } + } +} diff --git a/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ConformanceReportPlugin.java b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ConformanceReportPlugin.java new file mode 100644 index 000000000..ddb26765e --- /dev/null +++ b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ConformanceReportPlugin.java @@ -0,0 +1,422 @@ +package dev.openfeature.contrib.tools.providertck; + +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.ObjectMapper; +import edu.umd.cs.findbugs.annotations.SuppressFBWarnings; +import io.cucumber.plugin.ConcurrentEventListener; +import io.cucumber.plugin.event.EventPublisher; +import io.cucumber.plugin.event.Location; +import io.cucumber.plugin.event.Result; +import io.cucumber.plugin.event.Status; +import io.cucumber.plugin.event.TestCase; +import io.cucumber.plugin.event.TestCaseFinished; +import io.cucumber.plugin.event.TestRunFinished; +import io.cucumber.plugin.event.TestSourceRead; +import java.io.IOException; +import java.io.UncheckedIOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.InvalidPathException; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.ArrayList; +import java.util.Collections; +import java.util.Comparator; +import java.util.EnumMap; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.Set; +import java.util.function.Supplier; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * Writes a machine-readable conformance report for the suite that just ran. + * + *

Registered automatically by {@link AbstractProviderTckTest}, so an adopter changes nothing to + * get one. It is opt-in per run: set {@value #REPORT_DIR_ENV} (or the + * {@value #REPORT_DIR_PROPERTY} system property) and each suite writes + * {@code

/.json}. Emitting a report is a property of the run rather than of the + * code — CI asks for one, a developer running the suite locally does not — and unset means no + * report, which is not an error. Several suites in one JVM each write their own file, so flagd's two + * resolvers do not collide. + * + *

Why the per-scenario list is the load-bearing part. Appendix F requires that a + * scenario skipped for an undeclared capability is reported as skipped with the reason and never as + * passed. This plugin makes that checkable by a consumer rather than dependent on the runner's + * summary being trustworthy: it records the outcome of every scenario, exactly once, straight from + * Cucumber's own {@code TestCaseFinished} event. One event in, one entry out — there is no path by + * which a skip is also counted as a pass, which is precisely the bug the Go TCK had to fix, where + * the capability-skip signal did not reach the after-hook and every skipped scenario was recorded + * twice. + * + *

What identifies an entry. Feature and name are not enough: every row of a + * Scenario Outline shares one name, and the type-mismatch matrix in {@code errors.feature} is eleven + * rows. Each entry therefore also carries the Examples row it came from, resolved by + * {@link ScenarioExamples}, so a consumer can tell which row failed rather than keeping whichever it + * saw last. + * + * @see open-feature/spec#424 + */ +public final class ConformanceReportPlugin implements ConcurrentEventListener { + + /** Environment variable naming the directory reports are written to. */ + public static final String REPORT_DIR_ENV = "PROVIDER_TCK_REPORT_DIR"; + + /** + * System property naming the directory reports are written to, taking precedence over the + * environment variable. + * + *

Accepted in addition to {@value #REPORT_DIR_ENV} because {@code -D} is how a Maven or + * Gradle invocation is usually parameterised. The environment variable is the portable spelling + * and is what every other language's TCK reads, so a cross-language CI job can set one thing. + */ + public static final String REPORT_DIR_PROPERTY = "provider.tck.report.dir"; + + private static final Logger log = LoggerFactory.getLogger(ConformanceReportPlugin.class); + + private static final String LANGUAGE = "java"; + + private final List records = Collections.synchronizedList(new ArrayList<>()); + private final ScenarioExamples examples = new ScenarioExamples(); + private final Supplier reportDir; + private final Supplier> metadata; + + /** Creates the plugin Cucumber instantiates by name. */ + public ConformanceReportPlugin() { + this(ConformanceReportPlugin::configuredReportDir, TckRuntime::lastRunMetadata); + } + + ConformanceReportPlugin(Supplier reportDir, Supplier> metadata) { + this.reportDir = reportDir; + this.metadata = metadata; + } + + @Override + public void setEventPublisher(EventPublisher publisher) { + // Cucumber publishes the feature's own text before any scenario in it runs, which is where + // the Examples tables come from. Reading it here rather than resolving the feature file + // again keeps the report reading exactly the source the runner executed. + publisher.registerHandlerFor(TestSourceRead.class, read -> examples.read(read.getUri(), read.getSource())); + publisher.registerHandlerFor(TestCaseFinished.class, this::onTestCaseFinished); + // The end-of-run event carries the run's own result, which the report has no use for: what + // matters is the outcome of each scenario, which TestCaseFinished has already delivered. + publisher.registerHandlerFor(TestRunFinished.class, finished -> writeReport()); + } + + /** + * Returns the directory reports are written to, or {@code null} when none was configured. + * + * @return the configured report directory, trimmed, or {@code null} + */ + static String configuredReportDir() { + String property = System.getProperty(REPORT_DIR_PROPERTY); + if (property != null && !property.trim().isEmpty()) { + return property.trim(); + } + String environment = System.getenv(REPORT_DIR_ENV); + if (environment != null && !environment.trim().isEmpty()) { + return environment.trim(); + } + return null; + } + + private void onTestCaseFinished(TestCaseFinished event) { + TestCase testCase = event.getTestCase(); + Result result = event.getResult(); + records.add(new ScenarioRecord( + featureName(testCase), + testCase.getName(), + exampleOf(testCase), + Collections.unmodifiableList(new ArrayList<>(testCase.getTags())), + result.getStatus(), + messageOf(result), + result.getDuration().toNanos() / 1_000_000.0)); + } + + private void writeReport() { + String dir = reportDir.get(); + if (dir == null) { + return; + } + + Optional run = metadata.get(); + if (!run.isPresent()) { + // Nothing observed the suite, which means it never got as far as starting the runtime. + // The run has failed for some other reason by now; adding a report with an invented + // provider name on top of that would only mislead. + log.error( + "{} is set but no TCK suite ran, so there is nothing to report on. " + + "This normally means the suite failed before its backend stack started.", + REPORT_DIR_ENV); + return; + } + + write(build(run.get()), dir, run.get().configuration()); + } + + /** + * Assembles the report from what the run observed. + * + * @param run what the runtime recorded about this suite + * @return the report, ready to serialise + */ + ConformanceReport build(TckRunMetadata run) { + List observed; + synchronized (records) { + observed = new ArrayList<>(records); + } + observed.sort(Comparator.comparing((ScenarioRecord r) -> r.feature).thenComparing(r -> r.name)); + + Set declared = run.capabilities(); + List scenarios = new ArrayList<>(observed.size()); + + // Only a capability that something actually gated on can be said to have failed, so this + // counts the failures as the scenarios are converted rather than guessing afterwards. The + // count goes into the capability's reason, which the schema requires for anything that did + // not pass. + Map failed = new EnumMap<>(Capability.class); + + // And only a capability some scenario carries can be said to have been tested at all. A + // capability nothing in the suite exercises has no outcome to report — see capabilitiesOf. + Map exercised = new EnumMap<>(Capability.class); + + for (ScenarioRecord record : observed) { + scenarios.add(resolve(record, declared, exercised, failed)); + } + + return new ConformanceReport( + new ConformanceReport.Provider(run.providerName(), LANGUAGE, run.configuration()), + new ConformanceReport.Sdk(TckBuildInfo.SDK_NAME, TckBuildInfo.sdkVersion()), + new ConformanceReport.Tck( + TckBuildInfo.IMPLEMENTATION, + TckBuildInfo.tckVersion(), + TckBuildInfo.specRevision(), + TckBuildInfo.assetsTree()), + backendOf(run), + capabilitiesOf(declared, exercised, failed), + Collections.unmodifiableList(scenarios)); + } + + private static ConformanceReport.ScenarioResult resolve( + ScenarioRecord record, + Set declared, + Map exercised, + Map failed) { + Outcome outcome; + String reason; + + if (record.status == Status.PASSED) { + outcome = Outcome.PASSED; + reason = null; + } else if (record.status == Status.SKIPPED) { + outcome = Outcome.NOT_DECLARED; + reason = skipReason(record, declared); + } else { + outcome = Outcome.FAILED; + reason = record.message == null ? "the scenario was reported as " + record.status : record.message; + } + + // A scenario counts towards its capabilities only if it actually ran. Carrying the tag is + // not the same as exercising the capability: a scenario can carry two, and be skipped for + // the one this provider did not declare. Every scenario in events.feature is like that — + // the feature carries @events and each scenario adds @stale or @configuration-change — so + // counting by tag presence would report @events as passed for a provider that declared it + // and ran neither scenario. + if (outcome == Outcome.PASSED || outcome == Outcome.FAILED) { + for (Capability capability : gatingCapabilities(record.tags)) { + exercised.merge(capability, 1, Integer::sum); + if (outcome == Outcome.FAILED) { + failed.merge(capability, 1, Integer::sum); + } + } + } + + return new ConformanceReport.ScenarioResult( + record.feature, + record.name, + record.example, + record.tags.isEmpty() ? null : record.tags, + outcome, + reason, + record.durationMs); + } + + /** + * Explains a skip, preferring the capability the scenario needed over whatever was thrown. + * + *

Deriving the capability from the scenario's own tags rather than from the abort message + * keeps the two from drifting apart, and gives a consistent sentence across every language's + * TCK. The thrown message is the fallback, for a scenario skipped by something other than the + * capability gate. + */ + private static String skipReason(ScenarioRecord record, Set declared) { + for (Capability capability : gatingCapabilities(record.tags)) { + if (!declared.contains(capability)) { + return "requires capability " + capability.tag() + ", which this provider does not declare"; + } + } + return record.message == null ? "the scenario was skipped" : record.message; + } + + private static List gatingCapabilities(List tags) { + List gating = new ArrayList<>(tags.size()); + for (String tag : tags) { + Capability.fromTag(tag).ifPresent(gating::add); + } + return gating; + } + + /** + * Summarises each capability, with the reason the schema requires for anything but a pass. + * + *

This object covers the optional contract only, and is not a verdict on the provider: a + * scenario with no capability tag is mandatory and rolls up into nothing here, so a provider can + * fail one while every entry below reads {@code passed}. The per-scenario list is what a + * consumer has to read to decide whether a provider conforms. + * + *

A capability nothing exercised is left out. {@code @targeting} is + * reserved: it is in the tag vocabulary but no scenario carries it, because asserting that an + * evaluation context reached the backend needs an echo operation the control API does not have. + * A provider declaring it used to get {@code passed} for a claim nothing had examined — the + * vacuous pass the capability vocabulary exists to eliminate, arriving through the report rather + * than through the suite. The same pass arrives by a second route when every scenario carrying a + * declared capability was skipped for a different capability the provider did not + * declare, which is why exercising is counted by execution and not by tag presence. + * + *

Omitting is preferred to inventing a fifth outcome: the four in the schema describe what + * the provider did, and "nothing asked this of the provider" is a fact about the run. + */ + private static Map capabilitiesOf( + Set declared, Map exercised, Map failed) { + Map results = new LinkedHashMap<>(); + for (Capability capability : Capability.values()) { + ConformanceReport.CapabilityResult result; + if (!declared.contains(capability)) { + result = new ConformanceReport.CapabilityResult( + Outcome.NOT_DECLARED, + "not declared by this provider's configuration; the " + capability.tag() + + " scenarios were skipped and did not contribute to this result"); + } else if (!exercised.containsKey(capability)) { + // Declared, but no scenario carrying it ran — either nothing in the suite gates on + // it, or everything that does was skipped for some other capability this provider + // did not declare. Either way nothing was demonstrated, so there is nothing to + // report: a consumer sees the tag is absent rather than a pass it cannot rely on. + continue; + } else if (failed.containsKey(capability)) { + int count = failed.get(capability); + result = new ConformanceReport.CapabilityResult( + Outcome.FAILED, + count + " of " + exercised.get(capability) + " scenarios carrying " + capability.tag() + + " failed; the per-scenario results say which, and why"); + } else { + result = new ConformanceReport.CapabilityResult(Outcome.PASSED, null); + } + results.put(capability.tag(), result); + } + return Collections.unmodifiableMap(results); + } + + private static ConformanceReport.Backend backendOf(TckRunMetadata run) { + String description = run.backendDescription().orElse(null); + String controlApi = run.controlApi().orElse(null); + return description == null && controlApi == null + ? null + : new ConformanceReport.Backend(description, controlApi); + } + + @SuppressFBWarnings( + value = "PATH_TRAVERSAL_IN", + justification = "The directory is supplied by whoever started the test run, which is the " + + "whole point of the setting; the file name within it is sanitised by ReportNames") + private void write(ConformanceReport report, String dir, String configuration) { + Path directory; + try { + directory = Paths.get(dir); + } catch (InvalidPathException e) { + throw new IllegalStateException( + "provider-tck [" + configuration + "]: " + REPORT_DIR_ENV + " is not a usable path: " + dir, e); + } + Path path = directory.resolve(ReportNames.fileNameOf(configuration)); + + String json; + try { + json = new ObjectMapper().writerWithDefaultPrettyPrinter().writeValueAsString(report) + "\n"; + } catch (JsonProcessingException e) { + throw new IllegalStateException( + "provider-tck [" + configuration + "]: could not encode the conformance report", e); + } + + // A failure to write is raised rather than logged and swallowed. CI that asked for a report + // and silently did not get one is how a publishing pipeline serves a stale result forever. + try { + Files.createDirectories(directory); + Files.write(path, json.getBytes(StandardCharsets.UTF_8)); + } catch (IOException e) { + throw new UncheckedIOException( + "provider-tck [" + configuration + "]: could not write the conformance report to " + path, e); + } + + log.info("provider-tck [{}]: conformance report written to {}", configuration, path); + } + + private static String messageOf(Result result) { + Throwable error = result.getError(); + if (error == null) { + return null; + } + String message = error.getMessage(); + return message == null || message.trim().isEmpty() ? error.toString() : message.trim(); + } + + /** + * Returns the Examples row a test case came from, or {@code null} when it came from none. + * + *

Applies to every outcome, not only to failures. A row skipped for an undeclared capability + * is exactly as ambiguous as one that failed: eleven skips sharing a name say nothing about + * which eleven. + */ + private Map exampleOf(TestCase testCase) { + Location location = testCase.getLocation(); + return location == null ? null : examples.rowAt(testCase.getUri(), location.getLine()); + } + + /** Turns {@code classpath:features/errors.feature} into {@code errors}. */ + private static String featureName(TestCase testCase) { + String uri = testCase.getUri().toString(); + int lastSlash = uri.lastIndexOf('/'); + String base = lastSlash < 0 ? uri : uri.substring(lastSlash + 1); + int extension = base.lastIndexOf('.'); + return extension <= 0 ? base : base.substring(0, extension); + } + + /** One scenario as Cucumber reported it, before it is interpreted against declared capabilities. */ + private static final class ScenarioRecord { + private final String feature; + private final String name; + private final Map example; + private final List tags; + private final Status status; + private final String message; + private final double durationMs; + + ScenarioRecord( + String feature, + String name, + Map example, + List tags, + Status status, + String message, + double durationMs) { + this.feature = feature; + this.name = name; + this.example = example; + this.tags = tags; + this.status = status; + this.message = message; + this.durationMs = durationMs; + } + } +} diff --git a/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ControlApiClient.java b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ControlApiClient.java index 6c0a19e57..527e55876 100644 --- a/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ControlApiClient.java +++ b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ControlApiClient.java @@ -56,6 +56,19 @@ public String baseUrl() { return baseUrl; } + /** + * Returns how the backend was driven, for the conformance report. + * + *

{@code http} is the normative control API and the only one this TCK implements. The schema + * also allows {@code in-process}, a narrow allowance for providers with no backend at all; a + * report claiming it for a provider that has one should be treated with suspicion. + * + * @return the control API kind, always {@code http} + */ + public String controlApi() { + return "http"; + } + /** * Starts the backend with a named configuration, seeding flag state to that configuration's * baseline. diff --git a/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/Outcome.java b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/Outcome.java new file mode 100644 index 000000000..1fe055851 --- /dev/null +++ b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/Outcome.java @@ -0,0 +1,55 @@ +package dev.openfeature.contrib.tools.providertck; + +import com.fasterxml.jackson.annotation.JsonValue; + +/** + * The result of one scenario, or of one capability, in a conformance report. + * + *

There are four rather than two because "did not run" is not one thing. A capability the + * provider chose not to declare is a different statement from one the language makes impossible — + * {@code @strict-numeric-typing} cannot hold in a language with no integer type — and reporting both + * as "not declared" would show a whole language as missing something none of its providers can have. + * + *

The wire values are fixed by the report schema in the OpenFeature specification and are + * identical in every language's TCK. + */ +public enum Outcome { + + /** The scenario ran and every step passed. */ + PASSED("passed"), + + /** The scenario ran and a step failed. */ + FAILED("failed"), + + /** + * The scenario was not run because the provider did not declare the capability it needs. + * + *

Always carries a reason. A skipped scenario reported without one tells a reader that + * something was not checked but not what, which is barely better than omitting it. + */ + NOT_DECLARED("not-declared"), + + /** + * The scenario cannot apply to this provider, because the language makes it unsatisfiable. + * + *

Distinct from {@link #NOT_DECLARED} on purpose: nothing the provider author can do would + * change it, so a comparison page should not display it as a gap. + */ + NOT_APPLICABLE("not-applicable"); + + private final String value; + + Outcome(String value) { + this.value = value; + } + + /** + * Returns the value this outcome is written as in a report. + * + * @return the wire value defined by the report schema + */ + @JsonValue + public String value() { + return value; + } +} diff --git a/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ProviderTckHarness.java b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ProviderTckHarness.java index 1cc671faf..0c42fea71 100644 --- a/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ProviderTckHarness.java +++ b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ProviderTckHarness.java @@ -125,6 +125,23 @@ default Set capabilities() { return EnumSet.allOf(Capability.class); } + /** + * Returns the name of the provider configuration this suite exercises. + * + *

Used to name the conformance report — {@code

/.json} — and reported in + * it as the provider's configuration rather than its identity. The identity is what the provider + * says through its own metadata; this is which of its modes was tested, and a provider with two + * materially different modes produces two reports that are not interchangeable. + * + *

Derived from the suite class name by default: {@code FlagdInProcessTckTest} becomes + * {@code flagd-in-process}. Override it when that does not read well. + * + * @return a short name for this configuration + */ + default String configuration() { + return ReportNames.configurationOf(getClass()); + } + /** * Returns the Compose service name that hosts the control API and the backend the provider * connects to. diff --git a/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ReportNames.java b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ReportNames.java new file mode 100644 index 000000000..7f7e309a1 --- /dev/null +++ b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ReportNames.java @@ -0,0 +1,83 @@ +package dev.openfeature.contrib.tools.providertck; + +import java.util.Locale; + +/** + * Derives the names a conformance report is identified and filed under. + * + *

Kept apart from the report itself so that both the default + * {@link ProviderTckHarness#configuration()} and the file the plugin writes agree on one derivation. + */ +final class ReportNames { + + /** Suffixes a suite class name carries for JUnit's benefit rather than the report's. */ + private static final String[] SUITE_SUFFIXES = {"TckTest", "TCKTest", "TckSuite", "Test", "IT"}; + + /** Used when a name sanitises away to nothing, which an anonymous class manages. */ + private static final String FALLBACK = "provider-tck"; + + private ReportNames() {} + + /** + * Derives a configuration name from a suite class. + * + *

{@code FlagdInProcessTckTest} becomes {@code flagd-in-process}: the suffix that exists only + * so JUnit picks the class up is dropped, and the rest is hyphenated. A provider whose modes do + * not read well this way overrides {@link ProviderTckHarness#configuration()} and says so + * directly. + * + * @param suite the concrete suite class + * @return a hyphenated, lower-case configuration name + */ + static String configurationOf(Class suite) { + String simple = suite.getSimpleName(); + for (String suffix : SUITE_SUFFIXES) { + if (simple.length() > suffix.length() && simple.endsWith(suffix)) { + simple = simple.substring(0, simple.length() - suffix.length()); + break; + } + } + String hyphenated = simple.replaceAll("([a-z0-9])([A-Z])", "$1-$2") + .replaceAll("([A-Z]+)([A-Z][a-z])", "$1-$2") + .toLowerCase(Locale.ROOT); + return hyphenated.isEmpty() ? FALLBACK : hyphenated; + } + + /** + * Turns a configuration name into the file the report is written to. + * + *

Configuration names are chosen to read well in a failure message rather than to be + * path-safe, so anything that is not obviously safe becomes a hyphen. Without this a + * configuration named {@code flagd/rpc} would silently write outside the directory it was given. + * + * @param configuration the configuration name + * @return a file name ending in {@code .json} + */ + static String fileNameOf(String configuration) { + StringBuilder safe = new StringBuilder(configuration.length()); + for (int i = 0; i < configuration.length(); i++) { + char c = configuration.charAt(i); + boolean allowed = (c >= 'a' && c <= 'z') + || (c >= 'A' && c <= 'Z') + || (c >= '0' && c <= '9') + || c == '-' + || c == '_' + || c == '.'; + safe.append(allowed ? c : '-'); + } + String trimmed = trim(safe.toString()); + return (trimmed.isEmpty() ? FALLBACK : trimmed) + ".json"; + } + + private static String trim(String value) { + int start = 0; + int end = value.length(); + while (start < end && (value.charAt(start) == '-' || value.charAt(start) == '.')) { + start++; + } + while (end > start && (value.charAt(end - 1) == '-' || value.charAt(end - 1) == '.')) { + end--; + } + return value.substring(start, end); + } +} diff --git a/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ScenarioExamples.java b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ScenarioExamples.java new file mode 100644 index 000000000..2b5319a3c --- /dev/null +++ b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/ScenarioExamples.java @@ -0,0 +1,167 @@ +package dev.openfeature.contrib.tools.providertck; + +import io.cucumber.gherkin.GherkinParser; +import io.cucumber.messages.types.Envelope; +import io.cucumber.messages.types.Examples; +import io.cucumber.messages.types.Feature; +import io.cucumber.messages.types.FeatureChild; +import io.cucumber.messages.types.GherkinDocument; +import io.cucumber.messages.types.Scenario; +import io.cucumber.messages.types.TableCell; +import io.cucumber.messages.types.TableRow; +import java.net.URI; +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.Collections; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; +import java.util.stream.Stream; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * Resolves which Examples row a scenario came from, so the report can tell one row from another. + * + *

Every row of a Scenario Outline shares one scenario name. The type-mismatch matrix in + * {@code errors.feature} is eleven rows, so a report identified by feature and name alone carries + * eleven entries that nothing distinguishes: if one row fails and ten pass, the report cannot say + * which failed, and a consumer keying on feature and name keeps whichever it saw last. The + * parameters are the identity, so the report records them. + * + *

How the row is found. Cucumber's {@code TestCaseFinished} carries a + * {@link io.cucumber.plugin.event.TestCase}, and a test case is a compiled pickle that no longer + * knows it came from a table. What it does carry is a URI and a location, and for an outline-derived + * pickle that location is the Examples row: {@code TestCase.getLocation()} resolves the last + * of the pickle's AST node ids, which the pickle compiler sets to the {@code TableRow}. A plain + * scenario's last AST node is the scenario itself. So a line number in a feature file is enough to + * tell the two apart, provided the feature file is parsed as well as executed. + * + *

The source comes from Cucumber's own {@code TestSourceRead} event rather than from a file or a + * classpath resource. Cucumber has already located and decoded the feature — resolving + * {@code classpath:features/errors.feature} a second time would mean reimplementing that lookup, + * with a different answer whenever a consumer supplies features from somewhere else. The event is + * published before any scenario in that feature runs, so the table is always in place by the time a + * result needs it. + * + *

Parsing uses the Gherkin parser Cucumber already depends on, so the report reads the same + * document the runner executed rather than a second interpretation of the syntax. + */ +final class ScenarioExamples { + + private static final Logger log = LoggerFactory.getLogger(ScenarioExamples.class); + + /** + * Examples rows by feature URI, then by the line the row sits on. + * + *

Concurrent because {@link io.cucumber.plugin.ConcurrentEventListener} permits events from + * several threads. The TCK suite pins Cucumber to serial execution, but this class is not the + * place to depend on that. + */ + private final Map>> rowsByUri = new ConcurrentHashMap<>(); + + /** + * Parses a feature file and remembers every Examples row in it. + * + * @param uri the feature's URI, as Cucumber reports it + * @param source the feature file's text + */ + void read(URI uri, String source) { + if (uri == null || source == null) { + return; + } + + Map> rows = new LinkedHashMap<>(); + try (Stream envelopes = GherkinParser.builder() + .includeSource(false) + .includeGherkinDocument(true) + .includePickles(false) + .build() + .parse(uri.toString(), source.getBytes(StandardCharsets.UTF_8))) { + envelopes.forEach(envelope -> envelope.getGherkinDocument() + .flatMap(GherkinDocument::getFeature) + .ifPresent(feature -> collect(feature, rows))); + } catch (RuntimeException e) { + // Cucumber parsed this same source to produce the scenarios, so failing here means the + // two parsers disagree rather than that the feature is broken. Report the entries + // without their parameters rather than failing a run over the report's own metadata: + // the outcomes are still correct, and the warning says why the entries are ambiguous. + log.warn( + "provider-tck: could not parse {} for its Examples tables, so scenarios from that " + + "feature will be reported without the row they came from", + uri, + e); + return; + } + + rowsByUri.put(uri, Collections.unmodifiableMap(rows)); + } + + /** + * Returns the Examples row a scenario came from. + * + * @param uri the feature's URI, as Cucumber reports it + * @param line the line the test case reported as its location + * @return the row's parameters keyed by column header, or {@code null} for a scenario that did + * not come from a Scenario Outline + */ + Map rowAt(URI uri, Integer line) { + if (uri == null || line == null) { + return null; + } + return rowsByUri.getOrDefault(uri, Collections.emptyMap()).get(line); + } + + private static void collect(Feature feature, Map> rows) { + for (FeatureChild child : feature.getChildren()) { + child.getScenario().ifPresent(scenario -> collect(scenario, rows)); + child.getRule().ifPresent(rule -> rule.getChildren() + .forEach(ruleChild -> ruleChild.getScenario().ifPresent(scenario -> collect(scenario, rows)))); + } + } + + private static void collect(Scenario scenario, Map> rows) { + for (Examples examples : scenario.getExamples()) { + List headers = + examples.getTableHeader().map(ScenarioExamples::valuesOf).orElse(Collections.emptyList()); + if (headers.isEmpty()) { + continue; + } + for (TableRow row : examples.getTableBody()) { + Map parameters = parametersOf(headers, valuesOf(row)); + if (!parameters.isEmpty()) { + rows.put(row.getLocation().getLine().intValue(), Collections.unmodifiableMap(parameters)); + } + } + } + } + + /** + * Pairs a row's cells with the column headers, verbatim. + * + *

No coercion and no trimming beyond the parser's own: Gherkin has no types, so {@code "1"} + * is the string {@code 1} and turning it into a number would make the report say something the + * table did not. Keys are in column order, which is how the table reads. + * + *

A row with a different number of cells than the table has headers cannot occur — Gherkin + * rejects such a table before Cucumber compiles a pickle from it — but pairing only as far as + * the shorter of the two keeps a malformed document from throwing out of a reporting path. + */ + private static Map parametersOf(List headers, List cells) { + Map parameters = new LinkedHashMap<>(); + int paired = Math.min(headers.size(), cells.size()); + for (int i = 0; i < paired; i++) { + parameters.put(headers.get(i), cells.get(i)); + } + return parameters; + } + + private static List valuesOf(TableRow row) { + List values = new ArrayList<>(row.getCells().size()); + for (TableCell cell : row.getCells()) { + values.add(cell.getValue()); + } + return values; + } +} diff --git a/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/TckBuildInfo.java b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/TckBuildInfo.java new file mode 100644 index 000000000..bb5e18a0f --- /dev/null +++ b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/TckBuildInfo.java @@ -0,0 +1,135 @@ +package dev.openfeature.contrib.tools.providertck; + +import dev.openfeature.sdk.OpenFeatureAPI; +import java.io.IOException; +import java.io.InputStream; +import java.util.Properties; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * What this TCK build is, and which conformance artifacts it carries. + * + *

A conformance report has to say which questions were asked, not only what the answers were, and + * neither fact is available from the running code. The spec revision the packaged Gherkin came from + * is a property of the build — the artifacts are in the JAR, the repository they came from is not — + * so it is baked into a generated resource at build time and read back here. The SDK version is the + * opposite case: the TCK depends on a version range, so what a consumer actually ran + * against is only knowable at runtime. + * + *

Everything here degrades to {@code unknown} rather than throwing. A report that is slightly + * less identifiable is worth more than a test run that fails while tidying up after itself. + */ +final class TckBuildInfo { + + /** Which TCK implementation this is, in the form the report schema asks for. */ + static final String IMPLEMENTATION = "java-sdk-contrib/tools/provider-tck"; + + /** Maven coordinates of the SDK, reported without a version. */ + static final String SDK_NAME = "dev.openfeature:sdk"; + + /** Stands in for anything the build or the classpath could not tell us. */ + static final String UNKNOWN = "unknown"; + + private static final Logger log = LoggerFactory.getLogger(TckBuildInfo.class); + + /** Generated by the build from the module POM; see {@code src/main/resources-filtered}. */ + private static final String BUILD_PROPERTIES = "provider-tck-build.properties"; + + /** Written into every JAR by maven-archiver, and the most direct statement of what resolved. */ + private static final String SDK_POM_PROPERTIES = "META-INF/maven/dev.openfeature/sdk/pom.properties"; + + private static final Properties BUILD = loadBuildProperties(); + + private TckBuildInfo() {} + + /** Returns the version of this TCK module. */ + static String tckVersion() { + return property("tck.version"); + } + + /** Returns the open-feature/spec commit the packaged conformance artifacts came from. */ + static String specRevision() { + return property("spec.revision"); + } + + /** + * Returns the git tree ID of the conformance artifacts, or {@code null} when it is not recorded. + * + *

Null rather than {@code unknown} because the schema constrains this field to forty hex + * characters, so an unknown value has to be omitted rather than described. + */ + static String assetsTree() { + String tree = BUILD.getProperty("spec.assetsTree", "").trim(); + return tree.matches("[0-9a-f]{40}") ? tree : null; + } + + /** + * Returns the OpenFeature SDK version that was actually on the classpath. + * + *

Read rather than declared. The TCK depends on an SDK version range so that adopting it can + * never force an upgrade, which means the version is a property of the consumer's build; a + * hardcoded one would be a second place to be wrong. + */ + static String sdkVersion() { + Properties pom = load(SDK_POM_PROPERTIES); + if (pom != null) { + String version = pom.getProperty("version", "").trim(); + if (!version.isEmpty()) { + return version; + } + } + + // The Maven descriptor can be stripped, or the SDK can arrive from somewhere that is not a + // JAR at all — an IDE's output directory, say — so fall back to the manifest. + Package sdkPackage = OpenFeatureAPI.class.getPackage(); + String implementation = sdkPackage == null ? null : sdkPackage.getImplementationVersion(); + if (implementation != null && !implementation.trim().isEmpty()) { + return implementation.trim(); + } + + log.warn( + "Could not determine the OpenFeature SDK version from {} or from the package manifest; " + + "the conformance report will say '{}'", + SDK_POM_PROPERTIES, + UNKNOWN); + return UNKNOWN; + } + + private static String property(String key) { + String value = BUILD.getProperty(key, "").trim(); + return value.isEmpty() ? UNKNOWN : value; + } + + private static Properties loadBuildProperties() { + Properties properties = load(packagePath() + BUILD_PROPERTIES); + if (properties == null) { + log.warn( + "{} is missing from the TCK JAR, so a conformance report cannot identify the build " + + "that produced it. This means the resource filtering configured in the " + + "provider-tck POM did not run.", + BUILD_PROPERTIES); + return new Properties(); + } + return properties; + } + + private static String packagePath() { + return TckBuildInfo.class.getPackage().getName().replace('.', '/') + "/"; + } + + private static Properties load(String resource) { + ClassLoader loader = TckBuildInfo.class.getClassLoader(); + try (InputStream in = loader.getResourceAsStream(resource)) { + if (in == null) { + return null; + } + Properties properties = new Properties(); + properties.load(in); + return properties; + } catch (IOException e) { + log.warn("Could not read {} from the classpath", resource, e); + return null; + } + } +} diff --git a/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/TckRunMetadata.java b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/TckRunMetadata.java new file mode 100644 index 000000000..aa6553cc7 --- /dev/null +++ b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/TckRunMetadata.java @@ -0,0 +1,82 @@ +package dev.openfeature.contrib.tools.providertck; + +import java.util.Collections; +import java.util.EnumSet; +import java.util.Optional; +import java.util.Set; + +/** + * What a conformance report needs to know about the run, gathered as the run proceeds. + * + *

It exists because the two facts a report needs most are known at different times and are gone + * by the time it is written. The declared capabilities and the backend description come from the + * harness when the suite starts; the provider's own metadata name is only observable once a scenario + * has built a provider. Meanwhile Cucumber emits {@code TestRunFinished} — where the file is written + * — after {@code @AfterAll} has already torn the {@link TckRuntime} down. + * + *

So the runtime fills this in as it goes and keeps it after {@link TckRuntime#stop()}, and the + * report plugin reads it at the end. + */ +final class TckRunMetadata { + + private final String configuration; + private final Set capabilities; + private final String backendDescription; + private final String controlApi; + + private volatile String providerName; + + TckRunMetadata(String configuration, Set capabilities, String backendDescription, String controlApi) { + this.configuration = configuration; + this.capabilities = capabilities.isEmpty() + ? Collections.emptySet() + : Collections.unmodifiableSet(EnumSet.copyOf(capabilities)); + this.backendDescription = backendDescription; + this.controlApi = controlApi; + } + + /** Returns the suite name, which is the provider configuration under test. */ + String configuration() { + return configuration; + } + + /** Returns the capabilities the harness declared. */ + Set capabilities() { + return capabilities; + } + + /** Returns a short description of the backend stack, or empty when there is none. */ + Optional backendDescription() { + return Optional.ofNullable(backendDescription); + } + + /** Returns how the backend was driven, or empty when there is no control API. */ + Optional controlApi() { + return Optional.ofNullable(controlApi); + } + + /** + * Records what the provider called itself, as observed from a scenario that built one. + * + * @param name the provider's own metadata name + */ + void recordProviderName(String name) { + if (name != null && !name.trim().isEmpty()) { + providerName = name; + } + } + + /** + * Returns the provider's own metadata name, falling back to the configuration name. + * + *

The fallback covers a run in which no scenario ever built a provider — every scenario + * skipped, or the stack never came up. Reporting the suite name there is more useful than the + * empty string the schema would reject. + * + * @return the name to report as the provider's identity + */ + String providerName() { + String observed = providerName; + return observed == null ? configuration : observed; + } +} diff --git a/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/TckRuntime.java b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/TckRuntime.java index 82da5aa11..30efe639e 100644 --- a/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/TckRuntime.java +++ b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/TckRuntime.java @@ -41,10 +41,19 @@ public final class TckRuntime { private static TckRuntime instance; + /** + * What the most recent suite was, kept after {@link #stop()} for the conformance report. + * + *

Cucumber emits the end-of-run event that writes the report after {@code @AfterAll} + * has stopped the runtime, so the report would otherwise have nothing left to describe. + */ + private static volatile TckRunMetadata lastRunMetadata; + private final ProviderTckHarness harness; private final ComposeContainer compose; private final ControlApiClient controlApi; private final BackendEndpoint endpoint; + private final TckRunMetadata metadata; private TckRuntime(ProviderTckHarness harness, ComposeContainer compose) { this.harness = harness; @@ -53,6 +62,12 @@ private TckRuntime(ProviderTckHarness harness, ComposeContainer compose) { String baseUrl = "http://" + compose.getServiceHost(harness.backendService(), null) + ":" + compose.getServicePort(harness.backendService(), harness.controlPort()); this.controlApi = new ControlApiClient(baseUrl, harness.settleTime()); + this.metadata = new TckRunMetadata( + harness.configuration(), + harness.capabilities(), + "Docker Compose stack " + harness.composeFile().getName() + ", service " + harness.backendService(), + controlApi.controlApi()); + lastRunMetadata = this.metadata; } /** @@ -121,6 +136,27 @@ public BackendEndpoint endpoint() { return endpoint; } + /** + * Records what the provider under test calls itself. + * + *

A conformance report identifies the provider by its own metadata name rather than by the + * suite's, and only a scenario that has built one can say what that is. + * + * @param name the provider's metadata name + */ + public void recordProviderName(String name) { + metadata.recordProviderName(name); + } + + /** + * Returns what was observed about the most recent suite, for the conformance report. + * + * @return the metadata of the last suite to start, or empty if none has + */ + static Optional lastRunMetadata() { + return Optional.ofNullable(lastRunMetadata); + } + private static ComposeContainer startCompose(ProviderTckHarness harness) { File composeFile = harness.composeFile(); if (!composeFile.isFile()) { diff --git a/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/steps/ProviderSteps.java b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/steps/ProviderSteps.java index 8fe239414..a88a3aa9c 100644 --- a/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/steps/ProviderSteps.java +++ b/tools/provider-tck/src/main/java/dev/openfeature/contrib/tools/providertck/steps/ProviderSteps.java @@ -149,11 +149,13 @@ public void createProvider(String flavour) { state.provider = provider; state.domain = domain; state.client = api.getClient(domain); - log.info( - "Registered {} provider {} under domain {}", - flavour, - provider.getMetadata().getName(), - domain); + + // A conformance report identifies the provider by what it calls itself, not by the suite + // name, and this is the only place that knows it. + String providerName = provider.getMetadata().getName(); + runtime().recordProviderName(providerName); + + log.info("Registered {} provider {} under domain {}", flavour, providerName, domain); } /** diff --git a/tools/provider-tck/src/main/resources-filtered/dev/openfeature/contrib/tools/providertck/provider-tck-build.properties b/tools/provider-tck/src/main/resources-filtered/dev/openfeature/contrib/tools/providertck/provider-tck-build.properties new file mode 100644 index 000000000..0366d8a4a --- /dev/null +++ b/tools/provider-tck/src/main/resources-filtered/dev/openfeature/contrib/tools/providertck/provider-tck-build.properties @@ -0,0 +1,8 @@ +# Generated at build time by Maven resource filtering. Do not edit the copy in target/. +# +# Identifies the TCK build and the conformance artifacts it carries, for the +# tck section of a conformance report. The values come from the provider-tck +# POM; see the comment there for how the spec revision is maintained. +tck.version=${project.version} +spec.revision=${provider-tck.spec.revision} +spec.assetsTree=${provider-tck.spec.assets-tree} diff --git a/tools/provider-tck/src/test/java/dev/openfeature/contrib/tools/providertck/ConformanceReportPluginTest.java b/tools/provider-tck/src/test/java/dev/openfeature/contrib/tools/providertck/ConformanceReportPluginTest.java new file mode 100644 index 000000000..13eb2db41 --- /dev/null +++ b/tools/provider-tck/src/test/java/dev/openfeature/contrib/tools/providertck/ConformanceReportPluginTest.java @@ -0,0 +1,578 @@ +package dev.openfeature.contrib.tools.providertck; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.entry; + +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.ObjectMapper; +import io.cucumber.plugin.event.Event; +import io.cucumber.plugin.event.EventHandler; +import io.cucumber.plugin.event.EventPublisher; +import io.cucumber.plugin.event.Location; +import io.cucumber.plugin.event.Result; +import io.cucumber.plugin.event.Status; +import io.cucumber.plugin.event.TestCase; +import io.cucumber.plugin.event.TestCaseFinished; +import io.cucumber.plugin.event.TestRunFinished; +import io.cucumber.plugin.event.TestSourceRead; +import io.cucumber.plugin.event.TestStep; +import java.io.IOException; +import java.net.URI; +import java.nio.file.Files; +import java.nio.file.Path; +import java.time.Duration; +import java.time.Instant; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.EnumSet; +import java.util.HashMap; +import java.util.HashSet; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.Set; +import java.util.UUID; +import java.util.stream.Stream; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; +import org.opentest4j.TestAbortedException; + +/** + * The conformance report exists to make one rule checkable, so these tests check it. + * + *

Appendix F requires that a scenario skipped for an undeclared capability is reported as skipped + * with the reason and never as passed. The Go TCK shipped a version of this emitter that broke the + * rule silently — its runner did not deliver the capability-skip signal to the after-hook, so every + * skipped scenario was recorded twice, once correctly and once as passed. The equivalent hazard in + * Cucumber would be an event arriving more than once per scenario, or a hook's success masking the + * abort, so the totals and the per-scenario outcomes are asserted directly rather than assumed. + */ +class ConformanceReportPluginTest { + + private static final Set DECLARED = EnumSet.of(Capability.OBJECT, Capability.EVENTS); + + private static final ObjectMapper MAPPER = new ObjectMapper(); + + private static final URI IDENTITY_URI = URI.create("classpath:features/identity.feature"); + + /** + * A feature whose shape is the one the report has to cope with: an outline whose rows share a + * name across two Examples tables, a plain scenario alongside it, and an outline gated on a + * capability so that a skipped row can be checked as well as a passing one. + * + *

Line numbers are looked up from this text rather than written down, so editing it cannot + * silently make a test assert about the wrong row. + */ + private static final String IDENTITY_FEATURE = String.join( + "\n", + "Feature: Report identity", + "", + " Scenario: An unknown flag key returns the code default", + " Given nothing in particular", + "", + " Scenario Outline: Requesting the wrong type returns the code default", + " Given a -flag with key \"\" and a default value \"\"", + "", + " Examples: a string flag requested as something else", + " | key | requested | default |", + " | string-flag | Boolean | false |", + " | string-flag | Integer | 1 |", + "", + " Examples: a boolean flag requested as something else", + " | key | requested | default |", + " | boolean-flag | String | fallback |", + "", + " Scenario Outline: A gated outline", + " Given nothing in particular", + "", + " @stale", + " Examples: gated by a tag on this block alone", + " | mode |", + " | one |", + "", + " Examples: not gated", + " | mode |", + " | two |", + ""); + + private static final String OUTLINE = "Requesting the wrong type returns the code default"; + + private static final String GATED_OUTLINE = "A gated outline"; + + @Test + @DisplayName("every scenario appears exactly once, and a skipped one is never called passed") + void everyScenarioAppearsExactlyOnce(@TempDir Path dir) throws IOException { + JsonNode report = run(dir, defaultScenarios()); + + List scenarios = new ArrayList<>(); + report.get("scenarios").forEach(scenarios::add); + + assertThat(scenarios).hasSize(defaultScenarios().size()); + assertThat(scenarios).extracting(s -> s.get("name").asText()).doesNotHaveDuplicates(); + + Map byOutcome = new HashMap<>(); + for (JsonNode scenario : scenarios) { + byOutcome.merge(scenario.get("outcome").asText(), 1L, Long::sum); + } + assertThat(byOutcome).containsOnly(entry("passed", 2L), entry("failed", 1L), entry("not-declared", 2L)); + assertThat(byOutcome.values().stream().mapToLong(Long::longValue).sum()).isEqualTo(scenarios.size()); + } + + @Test + @DisplayName("a capability skip carries the reason it was skipped") + void aCapabilitySkipCarriesItsReason(@TempDir Path dir) throws IOException { + JsonNode report = run(dir, defaultScenarios()); + + JsonNode stale = scenarioNamed(report, "stale scenario"); + assertThat(stale.get("outcome").asText()).isEqualTo("not-declared"); + assertThat(stale.get("reason").asText()) + .isEqualTo("requires capability @stale, which this provider does not declare"); + assertThat(stale.get("tags")).extracting(JsonNode::asText).contains("@stale"); + } + + @Test + @DisplayName("every capability is reported, and only a declared one that failed reads as failed") + void everyCapabilityIsReported(@TempDir Path dir) throws IOException { + JsonNode capabilities = run(dir, defaultScenarios()).get("capabilities"); + + for (Capability capability : Capability.values()) { + JsonNode result = capabilities.get(capability.tag()); + assertThat(result) + .as( + "capability %s is missing; a capability is left out only when it is declared and " + + "no scenario exercises it, which is not the case here", + capability.tag()) + .isNotNull(); + + String state = result.get("state").asText(); + if (!DECLARED.contains(capability)) { + assertThat(state).isEqualTo("not-declared"); + } else if (capability == Capability.EVENTS) { + // The failing scenario carries @events, so the capability cannot read as passed. + assertThat(state).isEqualTo("failed"); + assertThat(result.get("reason").asText()) + .as("someone comparing providers wants to know how much failed before opening the detail") + .contains("1 of 1"); + } else { + assertThat(state).isEqualTo("passed"); + } + + // The schema requires a reason for anything that did not pass, because a bare tag with + // no explanation is not something a person comparing providers can act on. + if ("passed".equals(state)) { + assertThat(result.has("reason")).isFalse(); + } else { + assertThat(result.get("reason").asText()).isNotEmpty(); + } + } + } + + @Test + @DisplayName("a scenario that did not pass always says why") + void everyNonPassingScenarioSaysWhy(@TempDir Path dir) throws IOException { + for (JsonNode scenario : run(dir, defaultScenarios()).get("scenarios")) { + if ("passed".equals(scenario.get("outcome").asText())) { + assertThat(scenario.has("reason")).isFalse(); + } else { + assertThat(scenario.get("reason").asText()) + .as("scenario %s", scenario.get("name").asText()) + .isNotEmpty(); + } + } + } + + @Test + @DisplayName("the report carries everything the schema requires") + void theReportCarriesWhatTheSchemaRequires(@TempDir Path dir) throws IOException { + JsonNode report = run(dir, defaultScenarios()); + + assertThat(report.get("schemaVersion").asText()).isEqualTo("1"); + assertThat(report.get("provider").get("name").asText()).isEqualTo("My Provider"); + assertThat(report.get("provider").get("language").asText()).isEqualTo("java"); + assertThat(report.get("provider").get("configuration").asText()).isEqualTo("my-provider-rpc"); + assertThat(report.get("sdk").get("name").asText()).isEqualTo("dev.openfeature:sdk"); + assertThat(report.get("sdk").get("version").asText()).isNotEmpty(); + assertThat(report.get("tck").get("implementation").asText()).isEqualTo("java-sdk-contrib/tools/provider-tck"); + assertThat(report.get("tck").get("specRevision").asText()).hasSizeGreaterThanOrEqualTo(7); + assertThat(report.get("tck").get("assetsTree").asText()).matches("[0-9a-f]{40}"); + assertThat(report.get("backend").get("controlApi").asText()).isEqualTo("http"); + } + + @Test + @DisplayName("the SDK version is read from the classpath rather than declared") + void theSdkVersionIsRead() { + assertThat(TckBuildInfo.sdkVersion()) + .as("the OpenFeature SDK is on the test classpath, so its version must be discoverable") + .isNotEqualTo(TckBuildInfo.UNKNOWN) + .matches("\\d+\\.\\d+.*"); + } + + @Test + @DisplayName("nothing is written when no report directory is configured") + void nothingIsWrittenByDefault(@TempDir Path dir) throws IOException { + FakeEventPublisher publisher = new FakeEventPublisher(); + new ConformanceReportPlugin(() -> null, () -> Optional.of(metadata())).setEventPublisher(publisher); + publisher.emit(finished(scenario("evaluation", "a scenario", "@object"), Status.PASSED, null)); + publisher.emit(new TestRunFinished(Instant.now(), new Result(Status.PASSED, Duration.ZERO, null))); + + try (Stream written = Files.list(dir)) { + assertThat(written).isEmpty(); + } + } + + @Test + @DisplayName("the report is named after the configuration, safely") + void theReportIsNamedAfterTheConfiguration() { + assertThat(ReportNames.configurationOf(MyProviderRpcTckTest.class)).isEqualTo("my-provider-rpc"); + assertThat(ReportNames.fileNameOf("flagd-rpc")).isEqualTo("flagd-rpc.json"); + assertThat(ReportNames.fileNameOf("flagd/rpc")) + .as("a configuration name is chosen to read well, not to be path-safe") + .isEqualTo("flagd-rpc.json"); + assertThat(ReportNames.fileNameOf("../escape")).isEqualTo("escape.json"); + } + + @Test + @DisplayName("each row of a Scenario Outline carries the Examples row it came from") + void outlineRowsCarryTheirExample(@TempDir Path dir) throws IOException { + JsonNode report = run(dir, identityScenarios()); + + List rows = scenariosNamed(report, OUTLINE); + assertThat(rows) + .as("three rows ran, so three entries are expected, all under the one outline name") + .hasSize(3); + + assertThat(rows) + .extracting(row -> MAPPER.convertValue(row.get("example"), Map.class)) + .containsExactly( + exampleOf("key", "string-flag", "requested", "Boolean", "default", "false"), + exampleOf("key", "string-flag", "requested", "Integer", "default", "1"), + exampleOf("key", "boolean-flag", "requested", "String", "default", "fallback")); + } + + @Test + @DisplayName("cell contents are reported verbatim as strings, because Gherkin has no types") + void cellsAreNotCoerced(@TempDir Path dir) throws IOException { + JsonNode example = + scenariosNamed(run(dir, identityScenarios()), OUTLINE).get(1).get("example"); + + assertThat(example.get("default").isTextual()) + .as("\"1\" is what the table said; a report that turns it into a number says something else") + .isTrue(); + assertThat(example.get("default").asText()).isEqualTo("1"); + } + + @Test + @DisplayName("a scenario that is not an outline row carries no example") + void aPlainScenarioCarriesNoExample(@TempDir Path dir) throws IOException { + JsonNode plain = scenarioNamed(run(dir, identityScenarios()), "An unknown flag key returns the code default"); + + assertThat(plain.has("example")) + .as("the field is omitted rather than emitted empty; the schema requires at least one property") + .isFalse(); + } + + @Test + @DisplayName("a row skipped for an undeclared capability still says which row it was") + void aSkippedOutlineRowCarriesItsExample(@TempDir Path dir) throws IOException { + List rows = scenariosNamed(run(dir, identityScenarios()), GATED_OUTLINE); + + // Gherkin allows a tag on an individual Examples block, so two rows of one outline can + // differ in whether the capability gate stops them. Both rows must appear, and each must + // say which row it was: a skip that took its sibling with it would be invisible in the + // totals, and a skip that cannot name its row is as ambiguous as a failure that cannot. + assertThat(rows).hasSize(2); + assertThat(rows).extracting(row -> row.get("outcome").asText()).containsExactly("not-declared", "passed"); + assertThat(rows) + .extracting(row -> MAPPER.convertValue(row.get("example"), Map.class)) + .containsExactly(exampleOf("mode", "one"), exampleOf("mode", "two")); + } + + @Test + @DisplayName("feature, name and example together identify a scenario uniquely") + void scenariosAreUniquelyIdentified(@TempDir Path dir) throws IOException { + JsonNode report = run(dir, identityScenarios()); + + // Compared as tuples rather than as joined strings, so no separator can make two + // distinct entries look alike or one entry look like two. + List> identities = new ArrayList<>(); + List> namesOnly = new ArrayList<>(); + for (JsonNode scenario : report.get("scenarios")) { + String feature = scenario.get("feature").asText(); + String name = scenario.get("name").asText(); + String example = scenario.has("example") ? scenario.get("example").toString() : ""; + namesOnly.add(Arrays.asList(feature, name)); + identities.add(Arrays.asList(feature, name, example)); + } + + assertThat(new HashSet<>(namesOnly)) + .as("the premise of this test: feature and name alone are not unique in this report") + .hasSizeLessThan(namesOnly.size()); + assertThat(identities) + .as("a consumer keying on feature, name and example must not lose an entry") + .doesNotHaveDuplicates(); + } + + @Test + @DisplayName("a declared capability no scenario exercises is omitted rather than called passed") + void anUnexercisedCapabilityIsOmitted(@TempDir Path dir) throws IOException { + Set declared = EnumSet.of(Capability.OBJECT, Capability.TARGETING); + JsonNode capabilities = run(dir, declared, defaultScenarios()).get("capabilities"); + + assertThat(capabilities.has(Capability.TARGETING.tag())) + .as( + "%s is declared and no scenario carries it; the suite asked no question, so it has " + + "no answer, and a green result there is a pass nothing examined", + Capability.TARGETING.tag()) + .isFalse(); + + assertThat(capabilities.get(Capability.OBJECT.tag()).get("state").asText()) + .as("the omission must be specific, not a general failure to report capabilities") + .isEqualTo("passed"); + } + + @Test + @DisplayName("a declared capability whose every scenario was skipped is omitted too") + void aCapabilityWhoseScenariosAllSkippedIsOmitted(@TempDir Path dir) throws IOException { + // @events is declared; @stale is not. Every scenario in events.feature carries both — the + // feature is tagged @events and each scenario adds @stale or @configuration-change — so + // declaring @events alone runs none of them. Counting a capability as exercised because a + // scenario carried its tag would report @events as passed here, which is the same vacuous + // pass as a reserved capability arriving by a different route. + Set declared = EnumSet.of(Capability.OBJECT, Capability.EVENTS); + List scenarios = Arrays.asList( + finished(scenario("evaluation", "an object scenario", "@object"), Status.PASSED, null), + finished( + scenario("events", "a stale scenario", "@events", "@stale"), + Status.SKIPPED, + new TestAbortedException("Skipped: provider does not declare capability STALE"))); + + JsonNode capabilities = run(dir, declared, scenarios).get("capabilities"); + + assertThat(capabilities.has(Capability.EVENTS.tag())) + .as( + "%s was declared and no scenario carrying it ran, so nothing was demonstrated", + Capability.EVENTS.tag()) + .isFalse(); + assertThat(capabilities.get(Capability.OBJECT.tag()).get("state").asText()) + .isEqualTo("passed"); + assertThat(capabilities.get(Capability.STALE.tag()).get("state").asText()) + .as("the undeclared capability is still reported, with the reason it was not") + .isEqualTo("not-declared"); + } + + /** A suite whose name the default configuration derivation has to cope with. */ + private static final class MyProviderRpcTckTest {} + + /** + * The events the identity tests work from: the feature source Cucumber would publish, then one + * test case per compiled pickle, each located on the line it came from. + */ + private static List identityScenarios() { + return Arrays.asList( + new TestSourceRead(Instant.now(), IDENTITY_URI, IDENTITY_FEATURE), + finished( + identity("An unknown flag key returns the code default", " Scenario: An unknown flag"), + Status.PASSED, + null), + finished(identity(OUTLINE, "| string-flag | Boolean"), Status.PASSED, null), + finished( + identity(OUTLINE, "| string-flag | Integer"), + Status.FAILED, + new AssertionError("resolved 1 with no error code")), + finished(identity(OUTLINE, "| boolean-flag | String"), Status.PASSED, null), + finished( + identity(GATED_OUTLINE, "| one |", "@stale"), + Status.SKIPPED, + new TestAbortedException("Skipped: provider does not declare capability STALE")), + finished(identity(GATED_OUTLINE, "| two |"), Status.PASSED, null)); + } + + private static TestCase identity(String name, String marker, String... tags) { + return new FakeTestCase(IDENTITY_URI, name, Arrays.asList(tags), lineOf(marker)); + } + + /** Finds the 1-based line the given text sits on, so no test hard-codes a line number. */ + private static int lineOf(String marker) { + String[] lines = IDENTITY_FEATURE.split("\n", -1); + for (int i = 0; i < lines.length; i++) { + if (lines[i].contains(marker)) { + return i + 1; + } + } + throw new AssertionError("no line of the test feature contains " + marker); + } + + private static Map exampleOf(String... keysAndValues) { + Map example = new LinkedHashMap<>(); + for (int i = 0; i < keysAndValues.length; i += 2) { + example.put(keysAndValues[i], keysAndValues[i + 1]); + } + return example; + } + + private JsonNode run(Path dir, List events) throws IOException { + return run(dir, DECLARED, events); + } + + private JsonNode run(Path dir, Set declared, List events) throws IOException { + FakeEventPublisher publisher = new FakeEventPublisher(); + new ConformanceReportPlugin(dir::toString, () -> Optional.of(metadata(declared))).setEventPublisher(publisher); + events.forEach(publisher::emit); + publisher.emit(new TestRunFinished(Instant.now(), new Result(Status.FAILED, Duration.ZERO, null))); + + Path written = dir.resolve("my-provider-rpc.json"); + assertThat(written).exists(); + return MAPPER.readTree(Files.readAllBytes(written)); + } + + private static TckRunMetadata metadata() { + return metadata(DECLARED); + } + + private static TckRunMetadata metadata(Set declared) { + TckRunMetadata metadata = new TckRunMetadata("my-provider-rpc", declared, "a test double", "http"); + metadata.recordProviderName("My Provider"); + return metadata; + } + + /** + * The scenarios every test works from: two that pass, one that fails, two skipped for a + * capability the harness did not declare. + */ + private static List defaultScenarios() { + return Arrays.asList( + finished(scenario("evaluation", "a plain scenario"), Status.PASSED, null), + finished(scenario("evaluation", "an object scenario", "@object"), Status.PASSED, null), + finished( + scenario("events", "an event scenario", "@events"), + Status.FAILED, + new AssertionError("expected PROVIDER_READY")), + finished( + scenario("events", "stale scenario", "@events", "@stale"), + Status.SKIPPED, + new TestAbortedException("Skipped: provider does not declare capability STALE")), + finished( + scenario("lifecycle", "a lifecycle scenario", "@lifecycle"), + Status.SKIPPED, + new TestAbortedException("Skipped: provider does not declare capability LIFECYCLE"))); + } + + private static TestCaseFinished finished(TestCase testCase, Status status, Throwable error) { + return new TestCaseFinished(Instant.now(), testCase, new Result(status, Duration.ofMillis(12), error)); + } + + private static TestCase scenario(String feature, String name, String... tags) { + return new FakeTestCase(URI.create("classpath:features/" + feature + ".feature"), name, Arrays.asList(tags), 1); + } + + private static JsonNode scenarioNamed(JsonNode report, String name) { + for (JsonNode scenario : report.get("scenarios")) { + if (name.equals(scenario.get("name").asText())) { + return scenario; + } + } + throw new AssertionError("no scenario named " + name + " in the report"); + } + + private static List scenariosNamed(JsonNode report, String name) { + List matching = new ArrayList<>(); + for (JsonNode scenario : report.get("scenarios")) { + if (name.equals(scenario.get("name").asText())) { + matching.add(scenario); + } + } + return matching; + } + + /** Collects the plugin's handlers so a test can drive them directly. */ + private static final class FakeEventPublisher implements EventPublisher { + + private final Map, List>> handlers = new HashMap<>(); + + @Override + public void registerHandlerFor(Class eventType, EventHandler handler) { + handlers.computeIfAbsent(eventType, key -> new ArrayList<>()).add(handler); + } + + @Override + public void removeHandlerFor(Class eventType, EventHandler handler) { + handlers.getOrDefault(eventType, Collections.emptyList()).remove(handler); + } + + @SuppressWarnings("unchecked") + void emit(T event) { + for (EventHandler handler : handlers.getOrDefault(event.getClass(), Collections.emptyList())) { + ((EventHandler) handler).receive(event); + } + } + } + + /** The parts of a Cucumber test case the report reads, and nothing else. */ + private static final class FakeTestCase implements TestCase { + + private final URI uri; + private final String name; + private final List tags; + private final int line; + + FakeTestCase(URI uri, String name, List tags, int line) { + this.uri = uri; + this.name = name; + this.tags = tags; + this.line = line; + } + + @Override + public Integer getLine() { + return line; + } + + /** + * The pickle's location, which for an outline-derived test case is the Examples row rather + * than the {@code Scenario Outline} line. That is what Cucumber reports, and it is the only + * thing tying a compiled pickle back to the table it came from. + */ + @Override + public Location getLocation() { + return new Location(line, 1); + } + + @Override + public String getKeyword() { + return "Scenario"; + } + + @Override + public String getName() { + return name; + } + + @Override + public String getScenarioDesignation() { + return uri + ":" + line; + } + + @Override + public List getTags() { + return tags; + } + + @Override + public List getTestSteps() { + return Collections.emptyList(); + } + + @Override + public URI getUri() { + return uri; + } + + @Override + public UUID getId() { + return UUID.randomUUID(); + } + } +}