Skip to content

test: [expected failures] base fixtures needing implementation fixes - #163

Draft
leongdl wants to merge 1 commit into
OpenJobDescription:mainlinefrom
leongdl:conformance-base-expected-failures
Draft

test: [expected failures] base fixtures needing implementation fixes#163
leongdl wants to merge 1 commit into
OpenJobDescription:mainlinefrom
leongdl:conformance-base-expected-failures

Conversation

@leongdl

@leongdl leongdl commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Important

Expected-failure fixtures — do not expect these to pass. Every fixture in this PR is believed spec-correct and FAILS against at least one current reference implementation. They are parked in base/proposed/, which the conformance runner does not discover, so merging keeps the suite green. Per-fixture observed output, spec citation, and classification are in proposed/README.md. When a fix lands, promote the fixture up one directory unchanged.

Contents (8 fixtures)

The int64 template-validation acceptance family (5) — both implementations accept all of these at openjd check despite INT being a 64-bit signed integer:

  • default: 9223372036854775808 (2^63)
  • minValue / maxValue / allowedValues entry at 2^63
  • default: -9223372036854775809 (−2^63−1)

Note the supplied-value branch is already correct in openjd-rs (and a green fixture for it ships in the companion PR); this family covers the template-validation axes, where both implementations still accept out-of-range integers.

Type-name case-sensitivity gate (1)type: string with no extensions: must be rejected (case-insensitivity is EXPR-gated per schema §2). openjd-rs accepts it (unconditionally case-insensitive parsing); the Python CLI correctly rejects. Implementations diverge; Rust-side bug.

§1.2.1 merge widening (2) — a job template that widens an environment template's constraint is accepted by both implementations; Python's merge returns the intersection instead of rejecting. The spec states "each constraint must become more constrained." Either enforce the rule or respecify intersection semantics.

Companion to the green-fixture PR from branch conformance-base-gaps.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD-WITH-NITS (5/5 with the same caveat) — spec: §1.2.1 ("each constraint must become more constrained", Job Template processed last). The literal reading supports rejection and the README honestly offers the alternative (intersection-merge) reading — this is closer to a spec decision than a clean mandate; the fixture may be invalidated by a spec revision rather than an implementation fix. Service note: adoption flips CreateJob accept→reject for queue-environment parameter merging (BealineDataPlane grafts queue-env param definitions onto customer templates), and the un-scoped constraint-OMISSION case is the real-world hazard — gate promotion on the spec decision plus a service deprecation path. Matches sweep (fails both). Accept twin: mainline 1.2.1--constraint-narrowing-int-range.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD-WITH-NITS — spec: §1.2.1, STRING minLength axis. Same narrow-vs-intersection ambiguity and same queue-env merge hazard as the int-range twin; good isolation (default satisfies both ranges). Accept twin: mainline 1.2.1--constraint-narrowing-minlength.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD (5/5, one dedupe action) — spec: §2 (base type names case-sensitive; case-insensitivity is EXPR-gated per RFC 0007 — the BNF literal plus the §2 sentence's negative implication is solid). Matches sweep (openjd-rs accepts, Python rejects) and the README states both. Action: this duplicates PR #166's base/job_templates/proposed/2--type-lowercase-string.invalid.yaml — same construct, same defect, two paths; keep one (or differentiate on the type token).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD-WITH-NITS (majority; 1 NOT-GOOD dissent) — the defect is real (both implementations accept 2^63; matches sweep) and rejection is a safe data-integrity adoption (values that can't round-trip int64 storage). Dissent (spec-literalist): the base 2023-09 spec never bounds — int64 exists only in the EXPR doc — so a bignum implementation is arguably conformant and this is a SPEC QUESTION, not an implementation bug. Consensus action: keep parked, but reword the README classification to 'pending base-spec integer-bounds erratum' and make that erratum the promotion gate. Also: README cites accept twin jobs/2.3--int-value-above-int64-max.invalid.test.yaml as 'in the main suite' — it's on PR #161's branch (and per that PR's review, it should move HERE).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD-WITH-NITS — same family, negative bound (-2^63-1). Matches sweep; accept twin 2.3--int-default-int64-min.yaml in PR #161. Same spec-erratum caveat as the family.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD-WITH-NITS — same family, minValue axis; per-field decomposition justified (separate parse paths). Matches sweep; twin in PR #161. Inventory note: negative-side rejects (minValue/maxValue/allowedValues below -2^63) are uncovered — likely same code path, but note it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD-WITH-NITS — same family, maxValue axis. Service note: likeliest of the five to hit real templates (maxValue=2^63 as an infinity sentinel) — flag in any service change notice. Matches sweep; twin in PR #161.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD-WITH-NITS — same family, allowedValues axis; an unrepresentable allowedValue is dead weight, low blast radius. Matches sweep; twin in PR #161.

@leongdl

leongdl commented Aug 15, 2026

Copy link
Copy Markdown
Contributor Author

Quorum review (5 independent agents). 8 fixtures + README. Net: 2 clean GOOD (type-lowercase, and the widening pair is GOOD-WITH-NITS), 6 GOOD-WITH-NITS. The central contested point: the five int64 fixtures document behavior BOTH implementations exhibit (matches the sweep) and are safe data-integrity adoptions — but the base 2023-09 spec text never bounds (int64 lives only in the EXPR doc), so the spec-literalist reviewer voted all five NOT-GOOD as 'spec question misclassified as implementation bug'. Majority keeps them parked here, with the classification updated and a base-spec integer-bounds erratum filed as the promotion gate. Also: 2--type-lowercase duplicates PR #166's 2--type-lowercase-string (same construct, same defect) — dedupe across the two PRs. README quality is the best of the five expected-failures branches (genuine dual-implementation attribution).

Spec-correct fixtures that FAIL against current reference implementations: the int64 acceptance family (default/minValue/maxValue/allowedValues at 2^63, default at -2^63-1), the base type-name case-sensitivity gate, and 1.2.1 merge-widening. Parked in proposed/ (not discovered by the runner) so merging keeps the suite green; per-fixture output, spec citation, and classification in proposed/README.md. Promote each fixture unchanged when its fix lands. Companion to conformance-base-gaps.

Review: quorum-review fixes — restructured to kind-level proposed/
directories (<component>/<kind>/proposed/) so promotion is a mechanical
one-directory move; README split per kind and corrected: the int64 family
is now classified as a spec question gated on a base-spec integer-bounds
erratum (the base doc states no bound; a bignum implementation is
arguably conformant), with genuine per-implementation attribution;
adopted jobs/proposed/2.3--int-value-above-int64-max.invalid.test.yaml
from the base-gaps PR (openjd-rs rejects the supplied 2^63 but
openjd-model 0.11.x accepts it — verified both this session), replacing
the stale claim that it 'lives in the main suite'; noted the widening
fixtures' scheduler-merge blast radius and that the duplicate
type-lowercase pin in the param-types PR was dropped in favor of this
branch's copy.

Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
@leongdl
leongdl force-pushed the conformance-base-expected-failures branch from 1c1d3d1 to f5837f2 Compare August 15, 2026 19:26
@leongdl

leongdl commented Aug 15, 2026

Copy link
Copy Markdown
Contributor Author

Quorum-review fixes applied and pushed (rebased onto mainline 3069673):

  • Restructured to kind-level parking: 6 template fixtures → base/job_templates/proposed/, the 2 widening fixtures → base/jobs/proposed/ (promotion = move up one directory); README split per kind.
  • Adopted 2.3--int-value-above-int64-max.invalid.test.yaml from the base-gaps PR (test: Add 34 base 2023-09 conformance fixtures, strengthen 6 #161): openjd-rs rejects the supplied 2^63 but openjd-model 0.11.x create_job accepts it — both verified by execution this session — so it belongs in the inventory, not the live suite. (The old README claim that it "lives in the main suite" and is "rejected correctly" was true only of openjd-rs.)
  • int64 family reclassified: spec question first — the base 2023-09 doc never bounds <integer> (int64 exists only in the EXPR type table), so promotion is gated on a base-spec integer-bounds erratum; a bignum implementation is arguably conformant as the text stands.
  • Widening entries now note the scheduler-merge blast radius (queue-environment parameter grafting) as part of the spec-decision gate.
  • Noted the duplicate type-lowercase pin in test: [expected failures] parameter-type fixtures needing implementation fixes #166 was dropped; this branch's copy is the single source.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant