feat(cranelift): GHC calling convention

[playX18]

This PR adds LLVM’s GHC (ghccc) calling convention to Cranelift as CallConv::Ghc / CLIF ghc, for x86_64, aarch64, and riscv64.

Why?

People writing compilers for functional languages often want "register
pinning": keep runtime / VM state in fixed hardware registers, and jump
between functions instead of doing normal C calls. Haskell does this with
the STG machine and LLVM's ghccc. SML/NJ (and similar CPS systems) do
heap-allocated continuations and need something like jump-with-arguments.
Scheme compilers in CPS style want the same kind of thing.

In Cranelift today, enable_pinned_reg only gives you one reserved
register. tail helps with return_call, but it is still basically a
SysV-like ABI: it still builds native frames, and it does not pin a big
set of STG-style registers for you.

So this PR adds LLVM's GHC calling convention. Then you can map many
virtual machine registers to real regs through the callconv, with no
callee-saves, and use return_call between ghc functions.

Summary

• Mirror LLVM STG register pinning: args only in fixed regs, no callee-saves, no return values, no stack spill of overflow args (Unsupported).
• return_call allowed between ghc functions (supports_tail_calls); does not use Tail’s callee-pops-stack-args behavior.
• x86_64: Sp is %rbp, so ghc frames are FP-less (RSP-relative spills only). Entry from SysV/Rust that needs Sp must use a naked asm trampoline (documented in cranelift/docs/ir.md); Cranelift system_v cannot place Sp in %rbp while using it as FP.
• aarch64 / riscv64: Sp is not the hardware FP, so full STG arg lists work from system_v without asm.
• Unsupported on s390x/pulley.
• Exceptions not supported for ghc.

Test plan

• Verifier: filetests/filetests/verifier/ghc-abi.clif
• Precise-output: isa/{x64,aarch64,riscv64}/ghc.clif
• cargo run -p cranelift-tools -- test ./filetests/filetests/isa/*/ghc.clif ./filetests/filetests/verifier/ghc-abi.clif

[playX18]

Sorry, messed up the fork. Had to re-create branch.

[fitzgen]

FYI: I added an agenda item for discussing this feature to tomorrow's cranelift meeting

[cfallin]

@playX18 can you summarize (in a bit more detail) what your use-case for this is? I ask because it's not a trivial change to our ABI code -- this adds a bunch of parallel paths that are conditional on the new calling convention, +1.2kLoC overall, which is a lot of maintenance burden in critical code that we may need to update for security fixes, optimizations, new features, etc. in the future. If this serves a significant and widespread use-case that benefits many users, it may be worth it; but we'll want to weigh this not as a "nice to have" feature with a ready PR that we can merge and move on from, but rather, something with ongoing cost.

And if, for example, the desire is to pin more registers with args in a tail-call-between-basic-blocks style, as your description suggests: would an extension to an existing ABI, where we "keep counting beyond the limit" (i.e. SysV but we go beyond the 6 existing int regs on x86-64), do as well for you? That kind of thing would be a lot simpler to merge into the existing code. Or do you really need compat with the GHC ABI?

(Looking forward to discussing in the meeting tomorrow; the purpose of this comment is to fact-find beforehand)

[playX18]

@cfallin

@playX18 can you summarize (in a bit more detail) what your use-case for this is? I ask because it's not a trivial change to our ABI code -- this adds a bunch of parallel paths that are conditional on the new calling convention, +1.2kLoC overall, which is a lot of maintenance burden in critical code that we may need to update for security fixes, optimizations, new features, etc. in the future. If this serves a significant and widespread use-case that benefits many users, it may be worth it; but we'll want to weigh this not as a "nice to have" feature with a ready PR that we can merge and move on from, but rather, something with ongoing cost.

And if, for example, the desire is to pin more registers with args in a tail-call-between-basic-blocks style, as your description suggests: would an extension to an existing ABI, where we "keep counting beyond the limit" (i.e. SysV but we go beyond the 6 existing int regs on x86-64), do as well for you? That kind of thing would be a lot simpler to merge into the existing code. Or do you really need compat with the GHC ABI?

(Looking forward to discussing in the meeting tomorrow; the purpose of this comment is to fact-find beforehand)

My use-case is a Scheme implementation whose compiler lowers programs into CPS. Calls between compiled Scheme procedures are effectively jumps carrying a continuation and a set of VM values.

The current calling convention is approximately:

(ctx, callee, argc, arg0, arg1, arg2, arg3)

where ctx contains runtime state such as the allocation pointer, stack/runstack (not machine stack!) pointers for overflow arguments (arg4+ go onto this runstack), exception state, and GC metadata.
Each call is always a tail-call (aka jump) to either procedure or heap allocated continuation.

The important requirements for me are:

• more register arguments than the platform ABI normally provides;
• predictable, fixed argument-register assignment;
• efficient return_call/tail transfers between functions using the convention;
• no callee-saved-register preservation at every Scheme-to-Scheme transfer;

Compatibility with GHC callconv and LLVM's ghccc is not really necessary for me. I simply chose ghccc because it exists, is easy to find how it works in LLVM depths and is being utilized by some FP compilers that are not GHC with similar purposes: no callee saves, efficient tail calls, and a lot of effectively "pinned" registers even if compiler does not guarantee that.

An extended Cranelift-native ABI—roughly “SysV/AAPCS register assignment, but continue allocating from an explicitly defined larger register set”—would likely work for me, but callee-saves are sort of a problem: they increase the code size and somewhat decrease performance (Tail callconv currently requires frame pointer to always be available). On my compiler I have verified overall ~18% compiled code size reduction compared to Tail cc, and most importantly 5% to 15% performance increase (depending on benchmarks).

I am also happy to narrow the initial implementation to x86-64 and aarch64, which are my immediate targets, rather than adding three architectures at once.

So, to summarize: I need a register-heavy, tail-call-oriented internal ABI, but I do not specifically need GHC interoperability. I would be happy to rework this toward a smaller Cranelift-specific convention if that is the direction everyone prefers.