Bare fn stored into a ptr slot is not boxed; unbox_closure on it segfaults with no compile diagnostic

[paul-hammant]

Summary

Passing a bare named function to a ptr-typed parameter (or assigning it into a ptr struct field) does not box it into an _AeClosure. A later unbox_closure(...) + call(...) on that pointer then reads a .env field off a raw code address and segfaults at runtime. The compiler emits no diagnostic — it builds clean.

This is a soundness gap: unbox_closure is only valid on a value that was actually boxed (which happens automatically only when the value crosses an fn-typed boundary), but nothing prevents feeding it an unboxed bare-fn pointer.

Repro (segfaults)

struct Holder { cb: ptr }

my_handler(x: int) -> int { return x + 1 }

// ptr-typed param: bare fn is passed as a raw code pointer, NOT boxed
store_ptr(h: *Holder, f: ptr) { h.cb = f }

main() {
    h = heap.new(Holder)
    store_ptr(h, my_handler)      // no box happens here
    g = unbox_closure(h.cb)       // reads .env off a code address
    r = call(g, 41)               // -> signal 11
    print("result=${r}\n")
}

$ ae run repro.ae
Program crashed (signal 11: segmentation fault)

ae build produces no warning or error.

Working counterpart (correct)

Changing only the parameter type from ptr to fn makes the compiler autobox the bare function at the call site, and the same unbox_closure/call works:

store_fn(h: *Holder, f: fn) { h.cb = f }   // fn param -> autobox
...
store_fn(h, my_handler)                     // boxed here
g = unbox_closure(h.cb)
r = call(g, 41)                             // => 42

$ ae run safe.ae
result=42

So the boxing is keyed entirely on the static type of the parameter/slot the value flows through (fn boxes, ptr does not), and unbox_closure trusts its input unconditionally.

Why it matters

This is a natural pattern when storing heterogeneous handlers in ptr fields of lists/maps (routers, dispatch tables, callback registries — it surfaced while building a trailing-block HTTP router DSL). The ptr field is often necessary for storage in generic collections, so the developer stores a bare fn into ptr, later unbox_closurees it, and gets a silent segfault with no hint that the missing box is the cause.

Suggested fix (any one, roughly increasing effort)

1. Diagnose the sink: have unbox_closure(p) require that p's provenance is a boxed closure; error at compile time when a value known to be an unboxed bare-fn pointer reaches it.
2. Diagnose the source: warn when a bare named function is coerced into a ptr slot without an intervening box (the point where the .env is silently dropped), suggesting an fn-typed param or an explicit box_closure(...).
3. Make it safe by construction: autobox a bare fn whenever it is stored into a ptr slot that is later unboxed — i.e. box at the fn→ptr coercion the same way fn→fn does. (Costs an allocation even when not needed; (1)/(2) are probably preferable.)

At minimum, the silent-segfault-with-no-diagnostic combination should become a compile-time error or warning.

Environment

• Branch: main (repro on current tip)
• Discovered while reviewing a closure-heavy router DSL; _aether_box_closure/_aether_unbox_closure are emitted in compiler/codegen/codegen.c; box_closure/unbox_closure symbols registered in compiler/analysis/typechecker.c.

[nicolas-maman]

Tried the autoboxing option and it is the wrong one. Reporting the evidence so it does not get re-attempted.

Implicitly boxing a bare function into a _AeClosure slot at the call site regressed three tests: crypto_mlkem (SIGSEGV), http_server_ops, and http_middleware_d1. Bisected in a clean worktree of main to confirm the autobox commit was the cause, then reverted.

The reason is in crypto_mlkem: it passes a bare function address into a pointer slot on purpose and uses it as a raw address, not as a callable closure. Autoboxing rewrites exactly that value. So the transform cannot distinguish "user meant a closure and forgot to box" from "user meant the address", and it silently corrupts the second case, which is worse than the current segfault because it is not localised to the mistake.

That leaves options 1/2 from the issue, the diagnostic. The design problem is the same one under a different sign: the diagnostic must not fire on the legitimate raw-address use that crypto_mlkem demonstrates, so it needs the target slot's type to actually say "closure" rather than "pointer". That distinction is what the checker currently lacks, and it is the real work here.

The issue's own ranking (diagnostic first, autobox third) was right.