Skip to content

test: Add 34 base 2023-09 conformance fixtures, strengthen 6 - #161

Open
leongdl wants to merge 1 commit into
OpenJobDescription:mainlinefrom
leongdl:conformance-base-gaps
Open

test: Add 34 base 2023-09 conformance fixtures, strengthen 6#161
leongdl wants to merge 1 commit into
OpenJobDescription:mainlinefrom
leongdl:conformance-base-gaps

Conversation

@leongdl

@leongdl leongdl commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Summary

34 new base 2023-09 fixtures plus 6 strengthened, from a systematic coverage audit of the base spec. Full suite goes 655 → 687 passing, 0 failing (plus 2 FEATURE_BUNDLE_1 fixtures, 57 → 57 incl. the new ones).

Areas covered

  • Numeric boundaries (the audit's feat: import 2023-09 specification #1 finding: every string-length boundary was covered on both sides, every numeric boundary on neither): INT default/minValue/maxValue/allowedValues accepted at 2^63−1 and −2^63; FLOAT accepted at 1e308, rejected at 1e309/.inf/.nan; supplied INT value at 2^63−1 runs, at 2^63 rejected at job creation.
  • Inclusive boundaries: value == minValue/maxValue accepted for INT and FLOAT; length == minLength/maxLength for STRING — an implementation using > for >= passed the old suite.
  • onExit runs: verified there was previously no base job fixture containing an onExit action at all; marker-based fixture added.
  • Cancelation defaults: TERMINATE + notifyPeriodInSeconds rejected; cancelation with no mode rejected; format-string mode rejected without FEATURE_BUNDLE_1.
  • SimpleAction desugaring: scripts print their own argv, pinning the interpreter prefix and per-key file extension — previously wholly unasserted.
  • Reference grammar: bare {{MyParam}} and four malformed dotted segments rejected — the inner-segment parser was previously untested.
  • Cross-type misuse and closed objects: minValue on STRING, minLength on INT, unknown key in a parameter definition.
  • Gating: let rejected in base step template and step script; bash: SimpleAction key rejected without FEATURE_BUNDLE_1.
  • Post-resolution length: format-string job name resolving to exactly 128 chars accepted, 129 rejected — the normative "after resolution" check, previously only static-length-tested.
  • Strengthened (edits): six 3.4.3 associative-combination job fixtures gained off-diagonal forbidden: lines. Before this, an implementation computing a full cross product where the spec requires a zip passed all six.

Deliberately excluded

Cases blocked on spec defects recorded in the audit (descending-range semantics where the spec's own example table contradicts both implementations and four existing fixtures; <posintstring> gating; maxLength: 0) — those need spec decisions first, not fixtures.

Companion PR

Spec-correct fixtures that FAIL against current implementations (the int64 template-validation acceptance family, the base type-name case gate, §1.2.1 merge widening) are in the expected-failures PR (branch conformance-base-expected-failures).

34 new fixtures plus 6 strengthened: INT/FLOAT numeric boundary accepts, inclusive-boundary accepts, onExit-runs, cancelation defaults, SimpleAction argv desugaring, reference-grammar negatives, cross-type field misuse, EXPR/FEATURE_BUNDLE_1 gating from the base side, post-resolution name length, and off-diagonal forbidden lines added to six associative combination fixtures. Suite: 655 -> 687 passing. Spec-correct fixtures that FAIL against current implementations (the int64 acceptance family, type-name case gate, merge widening) are split to conformance-base-expected-failures.

Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
@leongdl
leongdl requested a review from a team as a code owner August 2, 2026 02:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant