Skip to content

fix: Restate RFC 0005 coercion as satisfaction then conversion - #175

Merged
mwiebe merged 1 commit into
OpenJobDescription:mainlinefrom
mwiebe:rfc0005-coercion-destinations
Aug 19, 2026
Merged

fix: Restate RFC 0005 coercion as satisfaction then conversion#175
mwiebe merged 1 commit into
OpenJobDescription:mainlinefrom
mwiebe:rfc0005-coercion-destinations

Conversation

@mwiebe

@mwiebe mwiebe commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Description of the change. What is being added or fixed?

This change was identified while working on the Rust implementation. The
coercion code was too complicated to understand and review properly,
and we evaluated how to improve it. The change was to make coercion
run as two steps — first check whether the result already satisfies
the target, then convert it if not. Working through that split surfaced
adjustments we wanted to make in the specification itself, which this
PR contains. Because EXPR isn't yet widely deployed in production,
we believe this is still a good time to make a change like this.

Each field in a template gives the expression it contains a target type
the type of result the field expects. If the expression's result has a
different type, it may be converted losslessly: an int result in a field
that expects a string becomes that string. RFC 0005's "Implicit Type
Coercion" section defines these conversions, and this PR updates that
section. The coercion applied while resolving function calls — what makes
1 + 2.0 promote the 1 to 1.0 — is a separate mechanism and is not
touched.

A target type can offer a choice: an optional string field accepts a string
or null, and a command-line argument accepts a string, a list of strings, or
null. The current wording has two gaps there:

  1. Each rule applies only when the target offers exactly one candidate of
    its kind (one plain type, or one list type). If it offers two — e.g.
    float | int | string, which several built-in functions accept — the
    spec doesn't prescribe what to do.
  2. Two rules can apply at once with no tie-break. A range expression 1-5
    can become the string "1-5" or the list [1, 2, 3, 4, 5], and for a
    target accepting both, the spec doesn't say which you get.

This amendment states the procedure implementations should follow:

  1. If the result already fits the target, use it unchanged. An int fits
    int | string, so it stays an int rather than becoming a string.
  2. Otherwise, try converting to each offered type in a spec-prescribed order,
    and use the first conversion that works.

The order the types are tried in is specified by a small table with
two rules behind it: a number prefers to stay a number before becoming text,
and conversions that always succeed (anything can become a string; any
string is a valid path) are tried last. Non-list types come before list types.
So 5 against float | string becomes 5.0, "5" against int | float
becomes 5 while "5.0" becomes 5.0, and 1-5 against "list of ints or string"
becomes the string "1-5" — settling gap 2. Trying each offered type settles
gap 1, makes the outcome identical across conforming implementations,
and guarantees that adding another accepted type to a field never breaks
a conversion that used to work; the old wording accidentally implied otherwise.

Other clarifications included:

  • Each rule carried its own "only when the target does not already include
    ..." condition. Step 1 covers all of those, so it is stated once.
  • Nothing converts into null. A null result is accepted by an optional
    field because it is already null; a string containing the text "null"
    stays a string.
  • Placeholder types from generic function signatures (T, T1, ...) are
    never valid conversion targets. That was already the rule; it now
    explicitly applies at any nesting depth, e.g. list[T1].
  • Some values are unknown while a template is validated, and are checked
    against these same rules at the type level. The spec now says what
    validation may claim about the eventual result: a type the target accepts,
    though not necessarily which of several offered types the actual value
    will land on, since that can depend on the value itself.

Two conversions are added to the spec's list — string → bool (the same
case-insensitive spellings as the explicit bool() conversion) and
string → range_expr. They are non-destructive parses that succeed only for
strings that unambiguously denote a value of the target type, in the same
spirit as string → int and string → float, and they slot directly into
the ordering principles: tried after the numeric parses, before the universal
path fallback. Cases the spec already pinned down are otherwise unchanged,
and previously unspecified ones are now defined. The wiki's
Expression Language page gets matching wording.


By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@mwiebe
mwiebe requested a review from a team as a code owner August 19, 2026 18:38
This change started on the implementation side: we identified that the
coercion code had become too complicated to understand and review properly,
and evaluated how we might improve it. The improvement was to restructure
coercion as two explicitly ordered steps — first check whether the result
already satisfies the target, then convert it if not. Working through that
split surfaced adjustments we wanted to make in the specification itself,
restated here. Because EXPR isn't yet widely deployed in production, we
believe this is still a good time to make a change like this.

The coercion rules applied the scalar rule only "when the target types have a
single scalar type (without counting `nulltype` or `list[T]`)" and the list
rule only when there was a single list type, prescribing no coercion at all
for a target with two or more candidates of the same shape. That leaves
reachable targets undefined: a target built from several candidate signatures
can carry two scalar candidates (`zfill`, `int`, `float`, and `bool` each have
a `float | int | string` parameter position), and implementations coerce there
rather than reporting an ambiguity. The RFC also listed `range_expr` → `string`
and `range_expr` → `list[int]` as rules whose conditions both hold for a
`list[int] | string` target, with no stated winner.

Restate the section in the two steps an implementation actually performs:

- Satisfaction. If the result's type already satisfies the target it is used
  unchanged. Spell out the relation, including that a union target needs one
  member satisfied and that `list[T]` is covariant in `T`, so `list[int]`
  satisfies `list[any]` and `list[int | string]`. Note it is directional and
  therefore not the symmetric matching used to bind type variables — using one
  for the other accepts a `list[T1]` target by binding `T1` and discarding the
  binding — and that a result's type is never itself a union, since union
  constraints on unresolved values are decomposed first.
- Conversion. Otherwise convert toward one of the target's destinations, a
  union contributing each member, first success winning. This replaces the
  single-candidate conditions and makes a union accept at least what each
  member accepts on its own.

Destinations are ordered non-list before list, and within each group by a
per-result-type preference table set by two principles: a value prefers to
stay within its own kind, so a number remains a number before it becomes
text, and a conversion that can fail is attempted before one that always
succeeds, since a universal fallback attempted first would make every
destination after it unreachable. So `int` prefers `float` over `string`;
`float` prefers `int` (exact wholes) over `string`; `string` prefers `int`,
then `float`, then the selective `bool` and `range_expr` parses, then `path`,
which every string trivially satisfies; and a list source orders list
destinations by its element type's preference, recursively. This makes the
choice fully deterministic — `5` against `float | string` is `5.0`, `"5"`
against `int | float` is `5` — where a first-draft of this rewrite had left
same-shape order unspecified, letting the same template produce different
jobs on different conforming implementations. The non-list-first level
resolves the `range_expr` overlap: against `list[int] | string` the result is
the canonical string `"1-5"`, whose cost does not depend on the range size.

Add `string` → `bool` (the same case-insensitive spellings as RFC 0006's
explicit `bool()` conversion) and `string` → `range_expr` to the conversion
list. Both are non-destructive parses that succeed only for strings that
unambiguously denote a value of the target type, in the same spirit as
`string` → `int` and `string` → `float`, and they slot directly into the
ordering principles — after the numeric parses, before the universal `path`
fallback — so `"true"` against a `bool | path` target is `Bool(true)`.

Since satisfaction runs first, the conversions no longer need their "when the
target types do not include ..." conditions; those were restating the first
step. State that `nulltype` is never a destination, so a `string` whose text
is `"null"` does not become `null`, and that the type-variable rule holds at
any nesting depth: an implementation must reject a `list` destination whose
element type mentions an unbound type variable rather than binding the
variable and discarding the binding.

Also sharpen the unresolved-value narrowing: against a union target the
constraint narrows to the union of every destination with a type-level rule,
rather than betting on any one of them, because the type level cannot see the
payload that decides which destination wins. The narrowed constraint thus
always satisfies the target and always describes the concrete result — an
`unresolved[float]` narrows to `unresolved[int | string]` against
`int | string`, covering both the 3.0 payload that lands on `int` and the 3.5
payload that falls through to `string`. For a non-union target exactly one
destination exists, so the constraint is exactly the type evaluation will
produce.

Matching user-facing language in the wiki's Expression Language page. The
openjd-rs implementation matches this text, with every stated example pinned
by a test.

Signed-off-by: Mark <399551+mwiebe@users.noreply.github.com>
@mwiebe
mwiebe force-pushed the rfc0005-coercion-destinations branch from b53c27f to aaaa197 Compare August 19, 2026 18:39
@mwiebe

mwiebe commented Aug 19, 2026

Copy link
Copy Markdown
Contributor Author

@uberware this PR is tweaking the EXPR RFC / spec slightly about coercion, would love your eyes and opinions on it if you are able to look. (I'm planning to merge this, and iterate on any follow-ups)

@mwiebe
mwiebe merged commit be0aefb into OpenJobDescription:mainline Aug 19, 2026
2 checks passed
@mwiebe
mwiebe deleted the rfc0005-coercion-destinations branch August 19, 2026 18:51
@uberware

uberware commented Aug 20, 2026

Copy link
Copy Markdown

Happy to — thanks for the ping. Read the merged text closely; the two-step split is a real improvement and I'm implementing against it rather than the old wording. Notes below, in descending order of how much I think they matter.

The satisfaction/conversion split resolves something I had implemented by accident.

My evaluator got the "never convert what the target already admits" outcome by asking the old negated-inclusion questions (intfloat when the target does not include int, and so on). That works case by case but it isn't a stated rule, so there was nothing to appeal to when a target offered two candidates — I simply rejected those. Stating satisfaction first, once, and letting conversion be the fallback makes the rejection I was doing visibly wrong, which is the useful kind of spec change.

1. Does a failed list destination charge operations for what it materialized?

This is the one question I'd most like pinned down. "A destination that fails is not an error so long as a later one succeeds", combined with element-wise list[T]list[U], means a list[string] of ["1", "x"] against a list[int] | list[path] target converts elements until "x" fails, then starts over on list[path]. §1.3.10 doesn't say whether the abandoned elements count against the operation limit. Both answers are defensible — charging is easier to implement and harder to game, not charging is easier to reason about — but implementations will differ silently, and the difference is observable to a template author only as an unexplained limit breach. My own differential corpus already carries 248 recorded operation-count divergences against the Rust crate, so this is exactly the class of thing that turns into a support question later.

2. stringbool and stringrange_expr widen template acceptance, not just value selection.

The other rules in the table pick between values an implementation already had to produce. These two make expressions valid in positions where they previously weren't — a format string resolving to "yes" now reaches a boolean field. That seems intended (it aligns implicit coercion with the explicit bool() spellings), but it's the one part of the rewrite that changes whether a template validates, so it's worth confirming out loud, and worth a conformance fixture or two if the suite doesn't have them yet. I'd be glad to write those.

3. Is the range_exprlist[int] destination now unreachable in practice?

Non-list-before-list makes list[int] unreachable whenever the target also offers string, and the table says as much. I can't find a template position in the base spec or RFCs 0006/0007 whose target offers list[int] without string, which would make the rule dead outside explicit list(). If that's right it's worth a note in the RFC saying so; if I've missed a position, I'd like to know which, because I'd be getting it wrong.

4. Unresolved narrowing — agreed, and worth keeping the invariant explicit.

Narrowing to the union of every destination rather than betting on one is the right call, and the sentence that satisfaction narrows to the source type ( unresolved[list[int]] against list[any] stays unresolved[list[int]] ) is the part I'd have got wrong from the old text. The stated invariant — the narrowed constraint always satisfies the target and always describes the concrete result — is what makes phase-1 validation sound, and I'd keep it stated as an invariant rather than left implicit in the examples.

Where my implementation stands, for whatever it's worth as a second data point.

Three gaps against the merged text, all of them me accepting less than the spec now requires: multi-candidate scalar union targets (rejected today), stringbool, and stringrange_expr. Nothing I accept today becomes invalid under the new rules, which made this a comfortable change to receive.


Two things unrelated to the rewrite, since I have your attention:

The corpus offer is still open, and coercion is where it's thickest.

I asked on OpenJobDescription/openjd-rs#291 where you'd want the files, but I posted that comment after that issue had already been closed, so I suspect it was never seen — my fault for replying into a closed thread. The offer stands unchanged: 1,063 differential cases with 133 adjudicated value divergences and 248 operation-count divergences, each with written reasoning, offered under MIT-0. Tell me where to put them (an issue on either repo, a PR, a gist) and they go out the same day. The second question from that comment also still stands: is conformance-tests/ covered by the specification license (CC BY-ND 4.0) or by the sample-code license (MIT-0)? That one blocks me from starting the conformance-test translation you asked about, because I don't know what I'd be contributing into.

Two housekeeping items you may not have linked up:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants