x64: Add bitwidth-specific newtypes for registers. (#13184)

This change addresses the kind of lowering mismatch that we saw in
CVE-2026-24116 ([advisory]): a case where an instruction lowering
reasons about a narrower value (e.g. a scalar `f64`) in a wider register
(e.g. a 128-bit XMM), and implicit conversions cause load-sinking to
happen in a way that leads to a larger-than-expected load from memory.

Background: in Cranelift in general, we have an "upper bits undefined"
invariant. That is, a 32/64-bit FP value can live in a 128-bit XMM
register, or 8/16/32-bit integer can live in a 64-bit GPR, with no
problem. The instruction lowering sequences generally deal fine with
this: e.g., an `add` instruction doesn't care what is in the upper bits
(garbage in, garbage out, and carry only flows from lower to upper, not
the other way).

However, load-sinking, designed to be as "automatic" as possible to
apply in all of the places where it is applicable, throws a wrench into
this: because we don't make a distinction about how large the actual
operand is, and many layers of the ISLE deal only in register-related
operand types (`RegMem`, `GprMem` or `XmmMem`), we can get an implicit
load of the *entire* register width when we had started with a
narrowly-typed value. This is a miscompile that can become serious if
paired with a user of Cranelift (e.g. Wasmtime) that relies on certain
characteristics of out-of-bound accesses to maintain a sandbox.

In this PR, we take the approach discussed in #12477: we add newtypes
for all *specific* bit-widths of operand, to make the distinction solely
at metacopmilation time (i.e., when the ISLE DSL itself is typechecked)
in the lowering patterns. The expected widths flow upward from the
auto-generated instruction constructors in the assembler layer. We use
specific types almost universally; in a few cases where helpers do a
dynamic dispatch on type, we have specific generic-to-specific-width
converters that *also* take the `ty` and assert that it has the correct
width. Then, finally and most importantly, the auto-conversions for load
sinking will only work (and will release-assert otherwise) if the
type-width of the corresponding `Value` matches.

As a result of doing this large refactor, three separate bugs in
sunk-load width were found; see the new filetests. They are:

- `bitselect` on scalar f32/f64 args could sink a load, and would use a
  SIMD instruction that could fold a load and do a 128-bit load. (Wasm
  lowering only uses `bitselect` with 128-bit args.)

- `swiden_low` and `uwiden_low` could fold a 128-bit load and do a
  64-bit load (i.e., could *narrow* the memory op) because the SSE4.1
  instruction takes a 64-bit source operand. (Wasm lowering inserts
  bitcasts that prevent load-sinking in this case.)

- `fcvt_from_sint` could likewise turn a 128-bit load into a 64-bit
  load. (Wasm lowering likewise foils the bug with bitcasts.)

I verified that none of them are reachable from Wasm, fortunately, so we
don't have a repeat of the above CVE.

Fixes #12477.

[advisory]:
GHSA-vc8c-j3xm-xj73

Author: cfallin

cranelift/codegen/meta/src/gen_asm.rs

@@ -19,6 +19,9 @@ fn include_inst(inst: &Inst) -> bool {
 }
 
 /// Returns the Rust type used for the `IsleConstructorRaw` variants.
+///
+/// RegMem operand types are width-tagged based on the DSL operand width
+/// (e.g. `rm8` → `GprMem8`, `xmm_m128` with `align` → `XmmMemAligned128`).
 fn rust_param_raw(op: &Operand) -> String {
     match op.location.kind() {
         OperandKind::Imm(loc) => {
@@ -32,7 +35,8 @@ fn rust_param_raw(op: &Operand) -> String {
         OperandKind::RegMem(rm) => {
             let reg = rm.reg_class().unwrap();
             let aligned = if op.align { "Aligned" } else { "" };
-            format!("&{reg}Mem{aligned}")
+            let bits = rm.bits();
+            format!("&{reg}Mem{aligned}{bits}")
         }
         OperandKind::Mem(_) => {
             format!("&SyntheticAmode")
@@ -243,7 +247,8 @@ pub fn generate_rust_macro(f: &mut Formatter, insts: &[Inst]) {
 }
 
 /// Returns the type of this operand in ISLE as a part of the ISLE "raw"
-/// constructors.
+/// constructors. RegMem operand types are width-tagged based on the DSL
+/// operand width; see [`rust_param_raw`].
 fn isle_param_raw(op: &Operand) -> String {
     match op.location.kind() {
         OperandKind::Imm(loc) => {
@@ -265,7 +270,8 @@ fn isle_param_raw(op: &Operand) -> String {
         OperandKind::RegMem(rm) => {
             let reg = rm.reg_class().unwrap();
             let aligned = if op.align { "Aligned" } else { "" };
-            format!("{reg}Mem{aligned}")
+            let bits = rm.bits();
+            format!("{reg}Mem{aligned}{bits}")
         }
     }
 }

cranelift/codegen/src/isa/x64/inst.isle

@@ -24,13 +24,25 @@ pub trait FromWritableReg: Sized {
 
 /// A macro for defining a newtype of `Reg` that enforces some invariant about
 /// the wrapped `Reg` (such as that it is of a particular register class).
+///
+/// Each `reg_mem` / `reg_mem_imm` entry can optionally carry:
+///   * `aligned:BOOL`  -- require aligned memory operands (used for SSE
+///     non-VEX encodings).
+///   * `size:BITS`     -- the bit width of the value the operand holds. This
+///     is purely a type-level distinction (a sized newtype is a different
+///     Rust type than an unsized one of the same "family"), used to prevent
+///     cross-width operand mix-ups at compile time.
 macro_rules! newtype_of_reg {
     (
         $newtype_reg:ident,
         $newtype_writable_reg:ident,
         $newtype_option_writable_reg:ident,
-        reg_mem: ($($newtype_reg_mem:ident $(aligned:$aligned:ident)?),*),
-        reg_mem_imm: ($($newtype_reg_mem_imm:ident $(aligned:$aligned_imm:ident)?),*),
+        reg_mem: ($($newtype_reg_mem:ident
+                    $(size:$size:literal)?
+                    $(aligned:$aligned:ident)?),*),
+        reg_mem_imm: ($($newtype_reg_mem_imm:ident
+                        $(size:$size_imm:literal)?
+                        $(aligned:$aligned_imm:ident)?),*),
         |$check_reg:ident| $check:expr
     ) => {
         /// A newtype wrapper around `Reg`.
@@ -146,6 +158,13 @@ macro_rules! newtype_of_reg {
             }
 
             impl $newtype_reg_mem {
+                $(
+                    /// The bit width of the value this operand holds. This is
+                    /// a type-level distinction only; the underlying `RegMem`
+                    /// does not record the width.
+                    pub const SIZE_BITS: u32 = $size;
+                )?
+
                 /// Construct a `RegMem` newtype from the given `RegMem`, or return
                 /// `None` if the `RegMem` is not a valid instance of this `RegMem`
                 /// newtype.
@@ -227,6 +246,13 @@ macro_rules! newtype_of_reg {
             }
 
             impl $newtype_reg_mem_imm {
+                $(
+                    /// The bit width of the value this operand holds. This is
+                    /// a type-level distinction only; the underlying
+                    /// `RegMemImm` does not record the width.
+                    pub const SIZE_BITS: u32 = $size_imm;
+                )?
+
                 /// Construct this newtype from the given `RegMemImm`, or return
                 /// `None` if the `RegMemImm` is not a valid instance of this
                 /// newtype.
@@ -294,12 +320,24 @@ macro_rules! newtype_of_reg {
 }
 
 // Define a newtype of `Reg` for general-purpose registers.
+//
+// The sized `GprMemN` / `GprMemImmN` variants indicate the width of
+// the register or memory operand that an instruction operates on; for
+// a memory operand, the width of the load or store in particular.
 newtype_of_reg!(
     Gpr,
     WritableGpr,
     OptionWritableGpr,
-    reg_mem: (GprMem),
-    reg_mem_imm: (GprMemImm),
+    reg_mem: (GprMem,
+              GprMem8  size:8,
+              GprMem16 size:16,
+              GprMem32 size:32,
+              GprMem64 size:64),
+    reg_mem_imm: (GprMemImm,
+                  GprMemImm8  size:8,
+                  GprMemImm16 size:16,
+                  GprMemImm32 size:32,
+                  GprMemImm64 size:64),
     |reg| reg.class() == RegClass::Int
 );
 
@@ -324,12 +362,38 @@ impl Gpr {
 }
 
 // Define a newtype of `Reg` for XMM registers.
+//
+// Specific-width newtypes are, as for the GPR case above, used to
+// distinguish the width of the operation particularly when a memory
+// load or store occurs.
 newtype_of_reg!(
     Xmm,
     WritableXmm,
     OptionWritableXmm,
-    reg_mem: (XmmMem, XmmMemAligned aligned:true),
-    reg_mem_imm: (XmmMemImm, XmmMemAlignedImm aligned:true),
+    reg_mem: (XmmMem,
+              XmmMem8          size:8,
+              XmmMem16         size:16,
+              XmmMem32         size:32,
+              XmmMem64         size:64,
+              XmmMem128        size:128,
+              XmmMemAligned                    aligned:true,
+              XmmMemAligned8   size:8          aligned:true,
+              XmmMemAligned16  size:16         aligned:true,
+              XmmMemAligned32  size:32         aligned:true,
+              XmmMemAligned64  size:64         aligned:true,
+              XmmMemAligned128 size:128        aligned:true),
+    reg_mem_imm: (XmmMemImm,
+                  XmmMemImm8          size:8,
+                  XmmMemImm16         size:16,
+                  XmmMemImm32         size:32,
+                  XmmMemImm64         size:64,
+                  XmmMemImm128        size:128,
+                  XmmMemAlignedImm                    aligned:true,
+                  XmmMemAlignedImm8   size:8          aligned:true,
+                  XmmMemAlignedImm16  size:16         aligned:true,
+                  XmmMemAlignedImm32  size:32         aligned:true,
+                  XmmMemAlignedImm64  size:64         aligned:true,
+                  XmmMemAlignedImm128 size:128        aligned:true),
     |reg| reg.class() == RegClass::Float
 );