RFC: Generator functions #8934

Summary

Add generator functions to ReScript: gen functions that compile to JavaScript function*, with yield, yield*, and async gen for async generators. They produce the Generator.t<'yield, 'return, 'next> and AsyncGenerator.t values the standard library already models (#8355).

let range = gen (start, stop) => {
  for i in start to stop - 1 {
    yield i
  }
}
// range: (int, int) => Generator.t<int, unit, 'next>

for x of range(0, 3)->Generator.asIterable {
  Console.log(x)
}

Motivation

• JavaScript parity. Generators are the last major JavaScript control-flow construct ReScript cannot express. Consuming them already works (for..of, for await..of, the Generator and Iterator modules); producing them requires %raw.
• Libraries built on generators. Structured concurrency and effect systems use generators as their programming model: Effection, redux-saga, Effect. None of them can be used from ReScript without writing the generator functions in JavaScript.
• Lazy sequences. Iterators over trees, paginated APIs and infinite sequences are most naturally written as generators.

The groundwork is in place: the iterator and generator types (#8355), for..of and for await..of, break/continue (#8348), and try..finally (#8931), which generators need because cancellation (return() on a generator, an early exit from for..of, halting an Effection task) reaches the generator only through finally blocks.

Syntax

let numbers = gen () => {
  yield 1
  yield 2
}

let all = gen () => {
  yield* numbers()   // delegates to another generator
  yield 3
}

let lines = async gen file => {
  let handle = await open_(file)
  try {
    while !handle->eof {
      yield await handle->readLine
    }
  } finally {
    await handle->close
  }
}

• gen functions. gen is a contextual prefix for arrow functions, like async: gen (x) => …, gen x => …, gen () => …, and async gen (x) => …. gen stays an ordinary identifier everywhere else, so existing code that uses gen as a name keeps working. gen (x) => … and gen x => … are syntax errors today.
• yield e yields a value. yield () yields unit (JavaScript's bare yield). yield binds like JavaScript's: yield a + b is yield (a + b).
• yield* e delegates to another generator, as in JavaScript.
• yield is a keyword only inside gen bodies, and an ordinary identifier elsewhere, including in normal functions nested in a generator, as in JavaScript. This keeps yield* unambiguous (it can't be read as a multiplication) without reserving a new keyword globally.

The arrow lookahead for async and gen must not apply in switch-guard position. Today | Some(x) if async(x) => 1 fails to parse, because async(x) => is taken as an async arrow. With gen this would break real code, since gen is a common function name. A function is never valid in a guard (guards are bool), so the parser can skip the lookahead there. That also fixes the async case (#8933).

Typing

Generator functions

A gen function's body is typed in a generator context holding a yield type 'y and a resume type 'n:

• Function: if the body has type 'r, with yield type 'y and resume type 'n in scope, then gen (x: 'a) => body has type 'a => Generator.t<'y, 'r, 'n>.
• Yield: inside a generator with yield type 'y and resume type 'n, if e : 'y, then yield e : 'n. The value of a yield expression is what the consumer passes to next(v) (Generator.nextValue).
• Delegation: in the same context, if g : Generator.t<'y, 'r2, 'n>, then yield* g : 'r2. The inner generator's yield and resume types must match the outer ones; its return value becomes the value of yield*.
• Plain iterables: if e : iterable<'y> or e : array<'y>, then yield* e : unit. That's what built-in iterables return in JavaScript (undefined).
• Outside a generator: yield and yield* are errors.

The context is entered by gen functions and reset by every other function boundary, the same way break/continue scoping works. It is not reset by finally blocks: yield inside finally is allowed (JavaScript allows it, and Effection relies on it for asynchronous cleanup), while break and continue out of finally stay rejected (#8931).

The return type is the type of the body's final expression, unit for a generator that just ends. When early return (#8656) is added, return e in a generator will also need e : 'r.

Async generators

async gen functions have type 'a => AsyncGenerator.t<'y, 'r, 'n>. Inside them, await is allowed, and yield e takes e : 'y, the same rule as in sync generators. To yield the result of a promise, write yield await p.

Caveat: in JavaScript, yield v in an async generator waits for v if it is a promise. A value of type promise<_> yielded directly would therefore reach the consumer already resolved, while the types say it's a promise. This is the same nested-promise problem promise<promise<_>> has today. Making it an error to yield a value whose type is a promise inside async gen would close it.

Why yield* matters

Every yield in a generator shares one resume type 'n. Generators that only produce values ('n is never used) are not affected. Generators that receive typed answers, such as sagas, would need 'n to be a union of all possible answers. yield* avoids that, because its value is the inner generator's return type, which is new for each call. That's the pattern typed-redux-saga uses in TypeScript, and the one Effection's whole API is built on:

// Effection bindings
type effect // internal; users never construct it
type operation<'a> = Generator.t<effect, 'a, unknown>

@module("effection") external sleep: int => operation<unit> = "sleep"
@module("effection") external spawn: (unit => operation<'a>) => operation<task<'a>> = "spawn"
@module("effection") external run: (unit => operation<'a>) => task<'a> = "run"

let main = run(gen () => {
  let task = yield* spawn(gen () => {
    yield* sleep(1000)
    "done"
  })
  yield* Task.join(task) // string
})

operation is an alias of Generator.t, although Effection's operations are iterables rather than generators. The binding can't express that difference yet (see Future extensions).

Compilation

• Lambda. gen sets a generator flag in function_attribute, next to async. yield and yield* become primitives, which, like await, always count as having side effects.

• Optimizations. Generator functions are never inlined, and their self-calls are never turned into loops. Calling a generator function creates a generator object rather than running the body.

• JavaScript. Generator functions are emitted as function* (async: async function*), never as arrows, since JavaScript has no generator arrows.

• yield and await must never move into a function the compiler creates. One such case exists today and has to be fixed first: partial application puts its arguments inside the new closure. With h used more than once,

  let h = add(await p, ...)

  compiles to let h = (extra, extra$1) => add(await p, extra, extra$1);, an await inside a non-async arrow, which is a JavaScript syntax error. It also evaluates the argument on every call of h. Arguments should be let-bound outside the closure, as the optional-argument path already does (Partial application moves its arguments into the generated closure #8932). A yield there would fail the same way.

Editor support (hover, completion), gentype (the TypeScript mapping for Generator.t exists since #8355) and reanalyze need the usual updates. Reanalyze's termination analysis must accept while true { yield x }, and its exception analysis has to decide whether yield can throw (Generator.throwError throws into the generator at a yield).

Future extensions

These are not part of this proposal and can be added later without breaking anything.

Typed resume values (@yieldResult)

Libraries whose runtime answers each yielded effect with a value of a known type, like redux-saga, could mark their effect type:

@yieldResult
type effect<'res>

@module("redux-saga/effects")
external call: ('args => promise<'res>, 'args) => effect<'res> = "call"

let saga = gen () => {
  let user = yield call(fetchUser, id) // user: user
}

With the marker, yield e : 'a whenever e : effect<'a>, the way await types promise<'a>. The library's runtime is trusted to answer correctly, just as the JavaScript runtime is trusted to resolve a promise<'a> with an 'a. Without the marker, libraries write one helper generator per effect kind that casts the resume value, and users delegate with yield*.

Iterables with a return type

iterable<'a> carries only the yield type, so yield* over a library iterable that isn't a generator (Effection's operations, Effect's effects) can't be given its result type honestly. A three-parameter iterable type, like TypeScript 5.6's Iterable<T, TReturn, TNext>, with Generator.t, IterableIterator.t and IteratorObject.t as private abbreviations of it, would allow yield* on any such iterable.

for..of over generators without asIterable

for x of g with g: Generator.t<…> currently fails ("expected iterable<'a>"), because Generator.t is a private abbreviation of iterable<'yield>. An implicit coercion in for..of (and in yield*'s plain-iterable rule) would remove the need for ->Generator.asIterable. This is independent of generator functions and could ship first.

Open questions

1. The name of the @yieldResult attribute.
2. Whether yielding a promise inside async gen should be an error.
3. Whether reanalyze's exception analysis should treat yield as possibly throwing.

Plan

1. Prerequisites, each a separate PR:
   • Partial application moves its arguments into the generated closure #8932: let-bind partial-application arguments outside the generated closure. This fixes await in partial applications today.
   • Switch guard calling a function named async fails to parse #8933: skip the arrow lookahead in switch guards. This fixes if async(x) => today.
   • Optionally: implicit coercion to iterable in for..of.
2. Generator functions: syntax, typing, Lambda and JS output, tests, tooling.
3. Async generators.
4. Later, as needed: @yieldResult, iterables with a return type.