Add native Apple Metal/MPS backend

[TBO22]

Summary

This PR adds an experimental native Metal/MPS backend for CTranslate2 on Apple Silicon. It is opt-in with -DWITH_MPS=ON and makes the existing C++ and Python APIs accept device="mps" without changing the default CPU, CUDA, or HIP behavior.

I started this port about six months ago because CTranslate2 could use Apple CPUs efficiently but had no path to the Apple GPU. The first versions could run a few operations, but real Marian and Whisper inference exposed missing operators, excessive synchronization, lifetime bugs, and poor batch-size-1 performance. I used those end-to-end workloads—not only large synthetic GEMMs—to guide the implementation in this PR.

What is included

• Device::MPS support in device parsing, dispatch, storage, synchronization, the CLI, and Python bindings
• ctranslate2.get_mps_device_count() and get_supported_compute_types("mps")
• Source builds on Apple Silicon with Metal, Metal Performance Shaders, and Foundation
• FP32, FP16, BF16, and signed INT8 execution; compute_type="auto" selects FP16
• A thread-local persistent Metal execution stream that batches commands, reuses compute/blit encoders, supports several command buffers in flight, and only waits at host-visible synchronization points
• Shared-memory Metal allocation with an ordered interior-pointer lookup and GPU-lifetime tracking for buffers and Objective-C objects
• Cached Metal pipelines, MPS matrix multiplication objects, TopK objects, and persistent output-major decode weights
• A specialized FP16 M == 1 GEMV path for autoregressive projections, including a fused bias/residual/activation epilogue
• MPSMatrix and custom GEMM paths for prefill, transposed inputs, batched GEMM, broadcast strides, odd dimensions, and tails
• GPU argmax and small TopK (k <= 8), with MPSMatrix TopK used for supported larger values
• Metal implementations for the common elementwise, reduction, normalization, rotary, copy/layout, gather, concat/split/tile, and Conv1D paths needed by Transformer, Marian, and Whisper inference
• BF16 storage with FP32 accumulation where required, plus INT8 quantize/GEMM/dequantize support
• Optional MPS profiling, GEMM-path logging, synchronization logging, and command-buffer tuning environment variables
• MPS correctness tests and standalone primitive/Marian benchmark tools

The dispatch changes also preserve non-MPS builds. In particular, FP16/BF16 dispatch is restricted to enabled GPU backends so MSVC does not instantiate unsupported CPU half-precision symbols in CUDA wheel builds.

Build and use

cmake -S . -B build-mps \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_POLICY_VERSION_MINIMUM=3.5 \
  -DWITH_MPS=ON \
  -DWITH_ACCELERATE=ON \
  -DWITH_MKL=OFF \
  -DWITH_RUY=ON \
  -DOPENMP_RUNTIME=NONE
cmake --build build-mps -j

import ctranslate2

translator = ctranslate2.Translator(
    "model",
    device="mps",
    compute_type="float16",
)

The minimum deployment target is macOS 11. MPS is not enabled in prebuilt wheels by this PR and currently has to be built from source.

Correctness and CI validation

Local validation and the reportable benchmarks were run on a MacBook Air with an Apple M1 (7-core GPU, 8 GB unified memory), arm64, macOS 26.5.2, using Release builds.

The CI configuration includes a dedicated real Apple Silicon MPS GPU job. It builds with -DWITH_MPS=ON, runs the MPS runtime suite, and then runs the normal CPU/shared suite from the same MPS-enabled build. It also builds the Python bindings, tests explicit device="mps" inference, and verifies that device="auto" remains on CPU so the experimental backend is opt-in.

The equivalent workflow passed on the self-hosted Apple M1 runner: 192 MPS C++ tests passed, 198 CPU/shared C++ tests passed with one existing CPU-only skip, and both Python MPS integration tests passed.

https://github.com/TBO22/CTranslate2/actions/runs/32951573494/job/98123905546

The upstream macos-15-xlarge job could not be scheduled because larger-runner billing is not enabled for the repository. No test process started in that failed job; this was a runner availability/billing issue rather than a test failure. The upstream workflow retains macos-15-xlarge, so the merged CI configuration does not depend on contributor-owned hardware.

MPS coverage includes FP32/FP16/BF16, INT8, odd sizes, tails, broadcast and nonzero batch strides, transposed GEMM, alpha/beta, interior offsets, dependent asynchronous dispatches, TopK tie behavior, Gather, Conv1D, and translation output comparison. git diff --check also passes.

Quality Validation

The reproducible evaluation scripts and methodology are under tools/benchmark/quality/. CPU used FP32 and MPS used FP16 with the same models, inputs, sample order, and decoding settings. Model loading and one warm-up inference were excluded from inference timing.

Benchmark	CPU quality	MPS quality	Output agreement	MPS speedup
LibriSpeech clean, 2,703 utterances, beam 5	2.9125% WER	2.9089% WER	99.63% normalized transcript agreement	1.93×
WMT14 En→De, 2,737 sentences, beam 4, batch 4	27.9152 BLEU	27.9332 BLEU	96.60% exact translation agreement	2.13×

Performance

Performance varies by model and sequence length, so this is not intended as a universal claim. One representative end-to-end result from the M1 test machine was a 12.52-second, batch-size-1 Whisper transcription/translation workload:

Backend	Compute type	Latency	Output
CPU	FP32	54.950 s	reference
MPS	FP16	15.531 s	exact text match

This is a 3.54x end-to-end speedup for that workload. The PR also includes tools/benchmark_mps_marian.py, which runs CPU and MPS in separate processes with warmups and repeated measurements across source lengths and beam sizes, and tests/benchmark_mps.cc for decode GEMM, vocabulary projection, prefill GEMM, TopK, and copy-heavy operations.

Current limitations

• Apple Silicon and source builds only; no MPS-enabled release wheel in this PR
• WITH_MPS cannot currently be combined with CUDA or HIP in one build
• FlashAttention, AWQ INT4, INT16 GEMM, distributed collectives, and packed/shifted-u8 INT8 GEMM are not implemented on MPS
• Metal kernels are currently compiled from source once and cached at runtime; a precompiled metallib can be added separately
• INT8 is supported for compatibility and reduced weight memory, but FP16 is normally faster for batch-size-1 decoding and remains the automatic default

AI-assisted development disclosure

I used Codex with GPT-5.6 as a pair-programming tool for repository navigation, repetitive implementation work, test scaffolding, diff review, and tracing build and lifetime failures. I did not treat generated suggestions as authoritative. I researched the execution model and kernel design myself, including how Apple MLX kernels differ from ggml/PyTorch and CUDA implementations and how Apple unified memory affects synchronization and ownership. I selected the design, ran the benchmarks, reproduced correctness failures, reviewed the changes, and remain responsible for the implementation and results in this PR.

I understand that this is a large, performance-sensitive change and am happy to respond to detailed review or split follow-up work where maintainers prefer a different boundary.

[nerln]

I built this branch on an M4 to check the numbers, since the ones in the description come from an M1. They hold, though the ratio is lower here.

Setup: Apple M4 (10 CPU / 10 GPU, 16 GB), macOS 26, Release build of apple-mps-backend at 2f6a066, CMake 4.1. Systran/faster-whisper-large-v3, beam 5, batch 1, three 30-second Whisper windows after a discarded warm-up window, each configuration in its own process.

The C++ suite gives 370 passed and 2 skipped (Conv1DDilation and Conv1DGroupNoBiasQuantized, both pre-existing CPU skips), matching what you report.

Backend	Compute type	Latency	vs CPU float32	vs CPU int8
CPU	float32	59.82 s	1.00x	0.84x
CPU	int8	50.53 s	1.18x	1.00x
MPS	float16	24.91 s	2.40x	2.03x
MPS	float32	33.93 s	1.76x	1.49x
MPS	int8	57.61 s	1.04x	0.88x

MPS float16 and MPS float32 both produced text identical to CPU float32 over the whole workload.

Three things came out of it that may be worth folding into the PR.

1. The documented build has no int8 on CPU. Following the cmake line in the description:

>>> ctranslate2.get_supported_compute_types("cpu")
{'float32'}

WITH_MKL=OFF is required on Apple Silicon and Accelerate registers only a float32 GEMM, so get_gemm_backend(INT8) returns NONE and compute_type="int8" raises instead of falling back. Anyone benchmarking the way the description documents can only use the float32 baseline. Adding -DWITH_RUY=ON restores int8 and produces the 2.03x column, which is what somebody moving from the published wheel would actually see:

# ctranslate2 4.8.1 from PyPI
>>> ctranslate2.get_supported_compute_types("cpu")
{'float32', 'int8', 'int8_float32'}

I would put -DWITH_RUY=ON in the documented command and quote both baselines.

2. -DWITH_RUY=ON does not configure on CMake 4. third_party/ruy/third_party/cpuinfo and its deps/clog declare cmake_minimum_required below 3.5, and -DCMAKE_POLICY_VERSION_MINIMUM=3.5 gets past it. The PR does not cause this, but it stops the build immediately once int8 is enabled, so a line in the build section would save the next person the hour it cost me.

3. MPS int8 is slower than CPU int8, 57.61 s against 50.53 s, and less than half the speed of MPS float16. compute_type="auto" picks float16, so the default path is fine. The catch is that int8 is what many Whisper users carry over from their CPU settings, and they would land on the slowest configuration in the build while believing they had asked for the fast one. A fallback to float16 on MPS would cover it, or a line under limitations.

I did not test Marian, batch sizes above 1, multi-GPU, or memory pressure on an 8 GB machine.

[TBO22]

@nerln
Thanks a lot for taking the time to build and test this on your M4. The benchmarks and validation were really helpful, especially confirming that the MPS output matches CPU float32.
I also appreciate the suggestions. I’ll update the build instructions for Ruy and CMake 4, and improve how int8 is handled on MPS.
Also, seeing it already integrated into Scriba was honestly great. Thanks again for testing it so thoroughly.

[jordimas]

Thanks a lot for your work

My initial feedback:

• Please add a dedicated CI job that builds with -DWITH_MPS=ON and runs the MPS test suite on a runner with a real Apple Silicon GPU. A build-only job is not enough here; we need runtime coverage for device="mps".
  • Please also run the normal CPU test suite on the same -DWITH_MPS=ON build, to catch regressions in shared dispatch, build flags, Python bindings, and compute type selection.
• Please include quality benchmarks in addition to latency benchmarks:
  • For Whisper, report WER on a known benchmark and compare CPU vs MPS with the same model/settings. The faster-whisper benchmark setup is a useful reference:
    https://github.com/SYSTRAN/faster-whisper/tree/master/benchmark
  • For translation, report BLEU using the existing translation benchmarks in this repository if they are still applicable.
• When CTranslate2 is built with -DWITH_MPS=ON, MPS should remain voluntary opt-in. Please make sure device="auto" does not select MPS by default while this backend is experimental

[TBO22]

@jordimas Thanks for the feedback. I have addressed the requested items in the latest update.

• Added a dedicated macos-15-xlarge CI job that builds with -DWITH_MPS=ON and runs the MPS tests on an Apple Silicon GPU.
• The normal CPU/shared C++ suite runs from the same MPS-enabled build.
• Added Python tests for explicit MPS inference and verified that device="auto" still selects CPU.
• Added reproducible WER and BLEU benchmarks under tools/benchmark/quality/.

Full evaluation results:

Benchmark	CPU	MPS	Speedup
LibriSpeech clean, 2,703 utterances, beam 5	2.9125% WER	2.9089% WER	1.93×
WMT14 en-de, 2,737 sentences, beam 4, batch 4	27.9152 BLEU	27.9332 BLEU	2.13×

CPU used FP32 and MPS used FP16 with identical models and decoding settings. Model loading and warm-up were excluded from inference timing. Exact commands, hardware details, and methodology are documented in the benchmark README.

[TBO22]

The macos-15-xlarge job could not start because larger-runner billing is unavailable for the upstream PR. I therefore ran the complete WITH_MPS=ON test job on my self-hosted M1 runner. The MPS runtime tests, CPU/shared suite, Python bindings, explicit MPS inference, and device="auto" CPU-selection tests all passed. The upstream workflow still retains macos-15-xlarge.

Successful run: https://github.com/TBO22/CTranslate2/actions/runs/32951573494/job/98123905546