feat: add a schema for machine-readable provider conformance reports - #425
Draft
aepfli wants to merge 5 commits into
Draft
feat: add a schema for machine-readable provider conformance reports#425aepfli wants to merge 5 commits into
aepfli wants to merge 5 commits into
Conversation
First step of the design discussed in #424: the report format, before any implementation emits one, so four languages do not invent four dialects. Three properties are deliberate and are the parts worth arguing about. Four outcomes, not two. not-declared and not-applicable are distinct, and collapsing them misrepresents a provider: @strict-numeric-typing is unsatisfiable in JavaScript because the language has no integer type, so reporting it as not-declared would show every JS provider as missing something no JS provider can have. Per-scenario results are required, and required to be complete. That is what makes the appendix's own rule -- a skipped scenario is never reported as passed -- checkable rather than aspirational. It is not hypothetical: godog's summary counts capability-gated skips as passes, so the Go runner satisfies the rule only in a separate log line. A runner with that flaw still produces a truthful report here. The spec revision sits inside the tck object rather than standing alone, because it is a property of the artifact that ran rather than an independent assertion, and it travels with a digest of the assets actually executed. The revision is a claim; the digest is the check that catches a locally edited vendored file. There is deliberately no aggregate score. A single percentage would reward declaring fewer capabilities, which inverts the incentive the suite exists to create. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueThanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
… digest The schema carried a `gherkinDigest`, a SHA-256 over the feature files and the canonical flag set, on the argument that `specRevision` is an assertion and a digest is a check. The check is worth having; computing it ourselves is not. It would have made every implementation carry a normative hashing algorithm -- agreeing on file selection, path form, ordering, framing and line-ending normalisation, the last of which matters because Go embeds committed copies that a Windows checkout converts to CRLF. Four implementations of that is a lot of surface for something git already computes. `git rev-parse <revision>:specification/assets/provider-tck` is the same idea with none of the specification. It has two properties the digest was reaching for. It is verifiable: a third party can recompute it from the recorded revision, so a revision recorded wrongly does not pass unnoticed. And it is stable across unrelated spec commits, so two runs that executed identical assets report the same value even when pinned to different commits -- which the commit SHA alone gets wrong, since it changes for every unrelated edit to the specification. The drift the digest was chiefly aimed at -- a vendored copy edited locally while the recorded revision still points at the original -- is in any case already handled where it arises. Java, Python and JavaScript read the assets straight from the submodule, so no second copy exists to diverge. Go has committed copies and a CI check that regenerates them and fails on any difference. It is a build-time problem with a build-time fix, and pushing it into the report format would have every implementation re-detect what its own CI already catches. The cost is that the tree covers the asset directory's documentation too, so a README edit changes it. That errs towards reporting two comparable runs as different rather than the reverse, which is the safe direction. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
This was referenced Aug 24, 2026
Draft
Draft
Three changes, all of them found by implementing the format rather than by reading it. Writing four emitters against this schema was always going to be the real review, and it was. A reason is now required whenever an outcome is not "passed", at both the capability and the scenario level. The schema previously said it was "required in practice", which is another way of saying not required: nothing enforced it, and a report of bare tags with no explanation would have validated. The reason is most of the value to a person comparing providers -- "does not support configuration-change events" is information, "@configuration-change: not-declared" is a shrug. knownDeviation moves to a shared definition and becomes available on a scenario as well as a capability, because the two are different claims. A capability-level deviation says the provider does not do X. A scenario-level one pins the deviation to the single case that fails, which is what an implementation marking one expected failure has to record -- the Python suite marks exactly one, for open-feature/python-sdk#619, and until now had nowhere to put it but prose in the reason string. The capabilities object now states plainly that it is not a verdict. Scenarios carrying no capability tag are mandatory, roll up into no capability, and are therefore invisible in that summary -- so a provider can fail a mandatory scenario while every entry reads "passed". The Python implementation produced exactly that report. A consumer deciding whether a provider conforms has to read the scenarios, which is the reason those are required and required to be complete, and the schema should say so where someone will read it rather than leaving it to be discovered. All reports emitted by the Go, Python and JavaScript implementations still validate, and the two new constraints were checked to reject a report that omits a reason rather than merely being decorative. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
A scenario entry was identified by feature and name, and every row of a Scenario Outline shares one name. The type-mismatch matrix is eleven rows, so a report carried eleven entries distinguished by nothing but a duration measurement. When one row fails and ten pass -- which is what the Python implementation actually produced -- the report cannot say which failed, and a consumer keying on feature and name keeps whichever row it happened to see last. An entry now carries the Examples row it came from, as the row's parameters keyed by column header, and identity is feature, name and example together. The parameters are the identity, and they come from the feature file rather than from any runner, which is why this is a field rather than a naming convention. Mandating a mangled name would have put a separator, an ordering and an escaping rule for values containing the separator into normative text that four implementations must reproduce byte-for-byte, where drift stays invisible until two reports quietly fail to line up. That was not a hypothetical risk. Before this field existed the three implementations had already diverged on exactly this point, each reasonably and none compatibly: Go emitted the bare scenario name for all eleven rows, Python appended pytest's example id, JavaScript used jest-cucumber's expanded title. The same row of the same feature file had three identities, which defeats the comparison the format exists to make possible. Values are the cell contents verbatim as strings, because Gherkin has no types. Coercing "1" to a number here would make the report disagree with the table it came from. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
aepfli
added a commit
to open-feature/go-sdk-contrib
that referenced
this pull request
Aug 24, 2026
A scenario entry in the conformance report was identified by feature and name. Every row of a Scenario Outline shares one name, so the type-mismatch matrix in errors.feature produced eleven entries that differed only in durationMs. If one row failed and ten passed, the report could not say which failed, and a consumer keying on feature and name kept whichever row it read last. Per the report schema, a scenarioResult now carries `example`: the Examples row it came from, keyed by column header, with the cells verbatim as strings. Gherkin has no types, so "1" stays "1" rather than becoming 1 -- the report says what the table said. It is present only for outline rows and omitted otherwise. godog hands a hook an already-expanded pickle, whose step text has the parameters substituted in and whose row is otherwise gone. What survives is AstNodeIds, whose last entry is the id of the Examples TableRow. The row is therefore recovered by parsing the embedded feature files a second time and indexing every TableRow by that id. Those ids come from a counter godog creates once per run and shares across the files it parses, so reproducing them means reproducing godog's parse -- same files, same order, pickle compilation in between. That coupling is not left to be trusted: a pickle that came from an outline and did not resolve fails the run, because quietly returning to the ambiguity this field exists to remove is worse than a build failure. The capability gate records its outcome before a scenario starts, so it fills the field in too. Four skipped rows of the @object outline are as ambiguous as four failed ones. The gate's own bookkeeping is keyed by pickle id rather than by scenario name for the same reason. Gherkin allows an Examples block to carry its own tags, so two rows of one outline can differ in whether the gate stops them; keyed by name, gating one row suppressed the after hook for every row and the rows that did run would have vanished from the report. gherkin/go/v26 moves from an indirect requirement to a direct one. It is the same module and version godog already builds against, so no dependency is added and no go.sum entry changes. Verified against the schema on open-feature/spec#425 with a Draft 2020-12 validator: every report the self-tests emit validates, the eleven matrix rows carry eleven distinct examples matching the feature file, and (feature, name, example) is unique across every scenario in a report. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
8 tasks
Two implementations asked independently whether the four languages should agree on the key order within an example object. They currently do not: Java emits Examples column order, Go sorts alphabetically because encoding/json sorts map keys when marshalling, and neither is wrong. A JSON object is unordered, so the answer is that order carries no meaning and implementations need not agree. Saying so explicitly is worth a sentence, because the alternative is three more implementations each deciding privately and someone eventually byte-comparing two reports and concluding they disagree when they do not. If byte-level reproducibility is ever required -- an attestation over the document is the obvious case -- that is a canonicalisation problem for the whole report, not something to solve inside one field. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
aepfli
force-pushed
the
feat/provider-tck-report-schema
branch
from
August 24, 2026 19:57
5adef5b to
56ed4f9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a JSON Schema for the machine-readable output of one provider conformance run.
Stacked on #423 (Appendix F), which defines the suite this reports on. Review that one first; this PR is only the report format.
Phase 1 of #424, which is the design discussion. This is the part of that discussion that seems settled enough to write down: what a report has to contain for two reports to be comparable. Collection and trust — how OpenFeature gets reports from providers it does not host, and how much a self-published report should be believed — stay open on the issue and are deliberately not addressed here.
The decisions worth arguing about
Four outcomes, not two.
passed,failed,not-declared,not-applicable. Collapsing the last two would misrepresent whole languages:@strict-numeric-typingis unsatisfiable in JavaScript because the language has no integer type, so reporting it asnot-declaredwould show every JavaScript provider as missing something no JavaScript provider can have.Per-scenario results are required, and required to be complete. This is what makes Appendix F's "skipped, with the reason — never as passed" checkable instead of aspirational. It matters concretely: godog's summary counts capability-skipped scenarios as passed (
29 scenarios (29 passed)with two skipped), so the Go runner's headline number already says something false. A runner with that flaw still produces a truthful report here, and that is the point — the rule stops depending on each runner's summary being trustworthy.No aggregate score. No percentage, no "27/29", no grade. A single number invites comparing providers that declared different capability sets, which is exactly the comparison the tag vocabulary exists to prevent. A consumer that wants one can compute it and own the choice of denominator.
specRevisionandassetsTreeare a self-checking pair, and git computes both.assetsTreeis the git tree object ID ofspecification/assets/provider-tck. It does two jobs a commit SHA alone cannot. It is checkable —git rev-parse <specRevision>:specification/assets/provider-tckmust reproduce it, so a revision recorded wrongly does not pass unnoticed. And it is stable across unrelated spec commits, so two runs that executed identical assets report the same value even when pinned to different commits, which is precisely the comparability question a consumer is asking; a commit SHA changes for every unrelated edit to the specification and would make those two runs look incomparable.An earlier revision of this PR specified a SHA-256 content digest instead. It was dropped as machinery we would have had to write four times: agreeing on file selection, path form, ordering, framing and line-ending normalisation — the last mattering because Go embeds committed copies that a Windows checkout converts to CRLF. The drift it chiefly targeted, a vendored file edited locally while the revision still points at the original, is already handled where it arises: Java, Python and JavaScript read the assets straight from the submodule so no second copy exists to diverge, and Go has committed copies plus a CI check that regenerates them and fails on any difference. It is a build-time problem with a build-time fix.
The tradeoff kept: the tree covers the asset directory's documentation, so a README edit changes it. That errs towards reporting two comparable runs as different rather than the reverse.
configurationis not optional in practice. One provider can produce several non-interchangeable reports. flagd's RPC and in-process resolvers differ in whether they emitPROVIDER_STALE, so a report keyed on provider name alone would have to pick one and misrepresent the other.knownDeviationdistinguishes "chose not to implement an optional feature" from "has a known bug, tracked here". The tag vocabulary alone cannot express that difference, and three of the four implementations have already needed it: flagd narrows a float to an integer in both Java and Go, Python needsxfail(strict=True)for open-feature/python-sdk#619, and the JavaScript multi-provider flattens every child error code toGENERAL. Each invented a local shape for the same idea.What is not here
The emitters. No implementation writes this format yet. That is phase 2, and it should follow rather than precede agreement on the shape — writing four emitters against a schema still under discussion would mean rewriting four emitters.
Closes nothing; #424 stays open for collection and trust.