backends/cuda: a torch reference backend — edge0-8b and edge0-35b generate text on an NVIDIA GPU - #19

Open

agourakis82 wants to merge 32 commits into

Edge0-AI:mainfrom

agourakis82:cuda-backend

Commits on Sep 11, 2026

backends/cuda: skeleton facade (torch reference impl) + EDGE0_BACKEND=cuda dispatch

Demetrios Chiuratto Agourakis
authored and
agourakis82
committed
850cd6f
docs: document running on NVIDIA GPUs via mlx[cuda12] (no code changes)

Demetrios Chiuratto Agourakis
authored and
agourakis82
committed
497c032
backends/cuda/io.py: implement load_model (dispatch verified against real checkpoints)

Demetrios Chiuratto Agourakis
authored and
agourakis82
committed
a4e1796
backends/cuda: fix dispatch + key-prefix bugs found against REAL checkpoints; add MoESpec instances

Demetrios Chiuratto Agourakis
authored and
agourakis82
committed
4d67e08
backends/cuda/quant.py: mark int4 packing as VERIFIED against real mx.quantize output

Demetrios Chiuratto Agourakis
authored and
agourakis82
committed
1c4ee13
backends/cuda/quant.py: follow mx.gather_qmm broadcasting; parity tests vs real MLX

agourakis82
and
claude
committed
0c89f90
backends/cuda, docs, tests: fix what the patch series got wrong

agourakis82
and
claude
committed
ac0053a
backends: make routing and sampling run on the torch backend, same answers as MLX

agourakis82
and
claude
committed
716d977
streaming: run StreamingSwitchGLU on the torch backend; fix hot-stack out-of-bounds gather

agourakis82
and
claude
committed
2740f7a
prerouter, adapters: reach the backend only through the facade

agourakis82
and
claude
committed
6aa6ee8
streaming/install: install streaming experts into transformers models

agourakis82
and
claude
committed
4c1fa30
backends/cuda/io.py: load_model loads the real MLX checkpoints

agourakis82
and
claude
committed
46a34aa
engine: import without MLX; refuse non-MLX backends up front

agourakis82
and
claude
committed
6e13276
docs/nvidia.md: what the torch backend has, how it is checked, what is left

agourakis82
and
claude
committed
9e16abb
backends/cuda: torch port of bailing_hybrid (edge0-8b), checked layer by layer against MLX

agourakis82
and
claude
committed
8fbe20f
engine: run the edge0-8b engine on the torch backend; bound its memory

agourakis82
and
claude
committed
8af2e6b
backends/cuda: torch port of qwen3_5_moe (edge0-35b); the 35b engine runs on torch

agourakis82
and
claude
committed
b0e4d4d
docs/nvidia.md: cuInit 999 on our Spark was a host device policy, not the driver

agourakis82
and
claude
committed
7ab23fc
backends/cuda: parameters never require grad (the prefill held its whole autograd graph)

agourakis82
and
claude
committed
2597d09
backends/cuda: run on an accelerator; load_model moves init-time buffers to DEVICE

agourakis82
and
claude
committed
750a5b9

Commits on Sep 12, 2026

backends/cuda: gather quantized embedding rows on an int32 view (CUDA has no uint32 index)

agourakis82
and
claude
committed
f6276a8
tests: check the qwen3_5_moe port against MLX on the real edge0-35b weights

agourakis82
and
claude
committed
0903837
docs/nvidia.md: edge0-35b runs on the GB10, on the real weights

agourakis82
and
claude
committed
af697c4
docs/nvidia.md: edge0-35b on the GB10, layer by layer against MLX

agourakis82
and
claude
committed
b2b7a8a

Commits on Sep 13, 2026

backends/cuda: optional dequantized-weight cache (2.5x a decode step on a GB10)

agourakis82
and
claude
committed
eb5ef48

Commits on Sep 14, 2026

Merge upstream main into cuda-backend

agourakis82
and
claude
committed
890f39e

Commits on Sep 15, 2026

cuda: stop paying a host sync per expert in gather_qmm

agourakis82
and
claude
committed
800c306

Commits on Sep 28, 2026

Merge remote-tracking branch 'origin/main' into cuda-backend
agourakis82
committed
8e532d1
tests: incremental staged stack matches on both backends

agourakis82
and
claude
committed
7e5656f

Commits on Oct 2, 2026

review: remote code opt-in, optional cuda extra, README install path

agourakis82
and
claude
committed
72d58fc
review: take() result shape, Bailing wrapper args, honest smoke tests

agourakis82
and
claude
committed
50317a2
Merge upstream main into cuda-backend (adopt the python/ layout)

agourakis82
and
claude
committed
5153694