feat(qwen35): add paged-attention monolithic decode-only AR foundation

README.md

@@ -348,6 +348,7 @@ When compression is on, the request path picks one of three modes automatically,
 | `--prefix-cache-slots N` | — | Live prefix-cache slot count |
 | `--kv-cache-dir <path>` | — | Persist prefix cache to disk |
 | `--kv-cache-budget N` | — | On-disk cache size cap |
+| `--paged-attention` | off | Exact 16-token block-table decode for single-device Qwen3.6-27B AR; see [paged attention](optimizations/paged_attention/README.md) |
 
 **Bounded KV residency (KVFlash)**
 

optimizations/paged_attention/README.md

@@ -0,0 +1,133 @@
+<p align="left">
+  <a href="../../README.md">← lucebox-hub</a>
+</p>
+
+<h1 align="center">Luce Paged Attention</h1>
+
+<p align="center">
+  <strong>Exact fixed-block K/V for long-context serving.</strong><br/>
+  Every sequence owns a block table; decode reads reusable 16-token physical blocks straight through it.<br/>
+  Ragged 8-request profile (<code>128K + 7x8K</code>) on one RTX 3090:
+  <strong>82% less K/V storage and 1.35x faster attention</strong>.
+</p>
+
+---
+
+## How it works
+
+- **Blocks.** Full-attention K/V rows live in fixed 16-token physical blocks in
+  one pool. Nothing is evicted and nothing is approximated — this is a layout
+  and allocation mechanism, not a residency policy.
+- **Block tables.** Each sequence owns a logical→physical map. Decode walks it,
+  so a sequence never has to occupy one contiguous K/V range, and the pool only
+  ever holds live, block-rounded tokens.
+- **One decode op.** `GGML_OP_PAGED_ATTN` on CUDA and HIP: D=256 GQA, F16/Q4_0/Q8_0
+  K/V. One warp covers all query heads of a K/V group per row loaded, and long
+  contexts split into partitions merged by a stable split-softmax.
+- **Resident metadata.** The block table and sequence length live in the
+  persistent target cache next to the pool, so a decode step uploads 4 bytes per
+  newly filled block instead of the whole table every token.
+- **Prefill is unchanged.** The prompt runs through the existing exact chunked
+  path against a freshly reset pool, so its layout is identity-mapped. Paging
+  begins at autoregressive decode.
+
+Not to be confused with [KVFlash](../kvflash/README.md): that one bounds how much
+context stays resident and evicts cold chunks. Paging keeps every token.
+
+## Usage
+
+```bash
+./server/build/dflash_server model.gguf --paged-attention --max-ctx 131072
+```
+
+## Compatibility
+
+| | |
+|---|---|
+| Architecture | `qwen35` — dense Qwen3.5 / Qwen3.6 |
+| Placement | one local CUDA or HIP device |
+| Attention | full only (`--fa-window 0`) |
+| K/V types | F16, Q4_0, Q8_0 |
+| Decode | autoregressive, one live sequence |
+| Block size | 16 tokens, fixed |
+
+**Rejected at startup** (exit 2, with the reason): `--draft`, `--ddtree`,
+`--target-devices`, `--target-shard-ipc-bin`, a non-zero `--fa-window`,
+`--prefill-compression`, and `DFLASH_KVFLASH`. The rules live with every other
+launch-admission rule in `check_feature_compatibility()`; which architecture and
+placement they are allowed on is one row in `model_capabilities.h`.
+
+**Disabled, not rejected:** prefix, prefill, and disk snapshots. Their format
+assumes contiguous K/V rows, so `--paged-attention` zeroes the caps and says so.
+
+**Why one sequence.** Qwen3.6 is hybrid: 48 of its 64 layers hold recurrent Gated
+DeltaNet state rather than an attention cache, and that state is backend-global
+today. Two live requests would mix it even with perfectly isolated K/V blocks.
+Making it sequence-indexed is step 2 below.
+
+## Numbers
+
+Attention ops only, Q4_0 K/V, Qwen dims (`D=256, Hq=24, Hkv=4`). Uniform batches
+of 1/2/4/8 sequences, as `paged ÷ contiguous` throughput — above 1.0x is paged
+winning.
+
+| Context | RTX 3090 (CUDA) | Radeon 8060S (HIP) |
+|---|---|---|
+| 4K | 1.10–1.20x | 4.28–5.60x |
+| 32K | 1.12–1.23x | 4.72–5.16x |
+| 128K | 1.23–1.97x | 5.00–6.42x |
+
+Ragged 8-request profile, `128K + 7x8K` on the RTX 3090. Paging stores only live,
+block-rounded tokens (188,416) instead of padding every sequence to 128K
+(1,048,576 slots):
+
+| | contiguous | paged |
+|---|---|---|
+| attention step | 3.007 ms | **2.235 ms** (1.35x) |
+| K/V, one layer | 1152 MiB | **207 MiB** |
+| K/V, 16 full-attn layers | 18.0 GiB | **3.23 GiB** (−82%) |
+
+Two caveats. The HIP ratios are inflated by a weak native HIP
+`flash_attn_ext_vec` baseline — don't read them as whole-model throughput. And
+small ragged batches are still behind (`4K + 7x256` measures 0.48x), because
+short sequences carry dead partitions sized by the longest sequence; the stream-K
+kernel in step 4 is what fixes that.
+
+## Roadmap
+
+1. **Done.** Decode-only `ggml_paged_attn`, block manager, CUDA/HIP integration,
+   benchmarks.
+2. **Scheduler groundwork.** Sequence-indexed DeltaNet/conv state, iteration-level
+   scheduling, dynamic decode batches.
+3. **Paged-aware chunk prefill.** Arbitrary block-table writes, multi-token
+   queries over preceding paged K/V, causal attention within the chunk.
+4. **Continuous admission and interleaving.** Token budgets, decode
+   prioritization, admission control, cancellation, prefill/decode interleaving,
+   plus the stream-K decode kernel for ragged batches.
+5. **Variable-length batched prefill.**
+6. **Prefix caching / CoW.** Shared prefix blocks with reference counting and
+   copy-on-write.
+
+## Benchmarks and tests
+
+```bash
+cmake --build "$BUILD_DIR" --target bench_paged_attention
+"$BUILD_DIR/bench_paged_attention" --context 131072 --k-type q4_0 --v-type q4_0
+```
+
+Each run compares one native padded-contiguous attention op against one paged op
+at `n_seq=1,2,4,8`, then adds a ragged 8-request row. Both layouts get identical
+logical Q/K/V and identical quantized K/V rows, and both must pass a
+double-precision CPU oracle before anything is timed. The CSV reports median step
+time, aggregate query throughput, exact K/V and metadata bytes, and each path's
+oracle error. It measures attention ops — not the full Qwen graph, HTTP
+scheduling, or continuous-batching throughput.
+
+`ctest -R paged` covers the host allocator plus the nine F16/Q4_0/Q8_0 K/V
+combinations, in both the partitioned and the forced-single-partition kernel
+paths.
+
+The fixed-block/block-table shape follows the
+[llama.cpp paged-attention discussion][llama-discussion].
+
+[llama-discussion]: https://github.com/ggml-org/llama.cpp/discussions/21961

server/CMakeLists.txt

@@ -284,6 +284,7 @@ add_library(dflash_common STATIC
     src/common/dflash_draft_graph.cpp
     src/common/dflash_draft_kv.cpp
     src/common/dflash_spec_decode.cpp
+    src/common/paged_kv_pool.cpp
     src/common/layer_split_backend.cpp
     src/common/layer_split_runtime.cpp
     src/qwen35/graph_builders.cpp
@@ -1032,6 +1033,15 @@ if(DFLASH27B_TESTS)
             ${CMAKE_CURRENT_SOURCE_DIR}/deps/llama.cpp/ggml/src)
         add_test(NAME kvflash_pool_sizing COMMAND test_kvflash_pool_sizing)
     endif()
+    if(EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/test/test_paged_kv_pool.cpp")
+        # Pure host-side allocator test: no model, ggml, or GPU is required.
+        add_executable(test_paged_kv_pool
+            test/test_paged_kv_pool.cpp
+            src/common/paged_kv_pool.cpp)
+        target_include_directories(test_paged_kv_pool PRIVATE
+            ${CMAKE_CURRENT_SOURCE_DIR}/src)
+        add_test(NAME paged_kv_pool COMMAND test_paged_kv_pool)
+    endif()
     if(EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/test/test_restore_delta.cpp")
         add_executable(test_restore_delta test/test_restore_delta.cpp)
         target_include_directories(test_restore_delta PRIVATE ${DFLASH27B_SRC_INCLUDE_DIRS})
@@ -1204,6 +1214,42 @@ if(DFLASH27B_TESTS)
             ${DFLASH27B_SRC_INCLUDE_DIRS})
     endif()
     endif()
+
+    if((DFLASH27B_GPU_BACKEND STREQUAL "cuda" OR
+        DFLASH27B_GPU_BACKEND STREQUAL "hip")
+       AND EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/test/test_paged_attention.cpp")
+        add_executable(test_paged_attention test/test_paged_attention.cpp)
+        target_link_libraries(test_paged_attention PRIVATE
+            ggml ${DFLASH27B_GGML_BACKEND_TARGET} ggml-base)
+        if(DFLASH27B_GPU_BACKEND STREQUAL "cuda")
+            target_link_libraries(test_paged_attention PRIVATE CUDA::cudart)
+        else()
+            target_link_libraries(test_paged_attention PRIVATE hip::host)
+        endif()
+        target_include_directories(test_paged_attention PRIVATE
+            ${CMAKE_CURRENT_SOURCE_DIR}/deps/llama.cpp/ggml/include
+            ${CMAKE_CURRENT_SOURCE_DIR}/deps/llama.cpp/ggml/src)
+        add_test(NAME paged_attention COMMAND test_paged_attention)
+        add_test(NAME paged_attention_direct
+                 COMMAND test_paged_attention --direct)
+        set_tests_properties(paged_attention_direct PROPERTIES
+            ENVIRONMENT "GGML_CUDA_PAGED_ATTN_FORCE_PARTITIONS=1")
+    endif()
+    if((DFLASH27B_GPU_BACKEND STREQUAL "cuda" OR
+        DFLASH27B_GPU_BACKEND STREQUAL "hip")
+       AND EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/test/bench_paged_attention.cpp")
+        add_executable(bench_paged_attention test/bench_paged_attention.cpp)
+        target_link_libraries(bench_paged_attention PRIVATE
+            ggml ${DFLASH27B_GGML_BACKEND_TARGET} ggml-base)
+        if(DFLASH27B_GPU_BACKEND STREQUAL "cuda")
+            target_link_libraries(bench_paged_attention PRIVATE CUDA::cudart)
+        else()
+            target_link_libraries(bench_paged_attention PRIVATE hip::host)
+        endif()
+        target_include_directories(bench_paged_attention PRIVATE
+            ${CMAKE_CURRENT_SOURCE_DIR}/deps/llama.cpp/ggml/include
+            ${CMAKE_CURRENT_SOURCE_DIR}/deps/llama.cpp/ggml/src)
+    endif()
 endif()
 
 if(DFLASH27B_SERVER)

server/deps/llama.cpp/ggml/include/ggml-rpc.h

@@ -8,10 +8,10 @@ extern "C" {
 
 #define RPC_PROTO_MAJOR_VERSION    3
 #define RPC_PROTO_MINOR_VERSION    6
-#define RPC_PROTO_PATCH_VERSION    4
+#define RPC_PROTO_PATCH_VERSION    5
 
 #ifdef  __cplusplus
-static_assert(GGML_OP_COUNT == 104, "GGML_OP_COUNT has changed - update RPC_PROTO_PATCH_VERSION");
+static_assert(GGML_OP_COUNT == 105, "GGML_OP_COUNT has changed - update RPC_PROTO_PATCH_VERSION");
 #endif
 
 #define GGML_RPC_MAX_SERVERS       16

server/deps/llama.cpp/ggml/include/ggml.h

@@ -613,6 +613,8 @@ extern "C" {
         // remain stable within a protocol generation.
         GGML_OP_MUL_MAT_GROUPED_SRC,
 
+        GGML_OP_PAGED_ATTN,
+
         GGML_OP_COUNT,
     };
 
@@ -2458,6 +2460,30 @@ extern "C" {
             float                 scale,
             float                 alpha);
 
+    // Decode-only attention over independently allocated physical KV blocks.
+    // q:            [D, n_seq, n_head] F32 (one query per sequence)
+    // k/v:          [D, pool_tokens, n_head_kv]
+    // block_table:  [max_blocks, n_seq] I32
+    // kv_seq_lens: [n_seq] I32 (valid cached K/V tokens per sequence)
+    // max_kv_seq_len: host-known maximum of kv_seq_lens, used to size the
+    //                 CUDA/HIP launch without reading block-table capacity
+    // res:          [D, n_seq, n_head] F32
+    //
+    // A logical token t is read from physical row
+    // block_table[t/block_size, seq]*block_size + t%block_size.
+    // The initial CUDA/HIP implementation supports D=256 and independently
+    // typed F16, Q4_0, or Q8_0 K/V pools.
+    GGML_API struct ggml_tensor * ggml_paged_attn(
+            struct ggml_context * ctx,
+            struct ggml_tensor  * q,
+            struct ggml_tensor  * k,
+            struct ggml_tensor  * v,
+            struct ggml_tensor  * block_table,
+            struct ggml_tensor  * kv_seq_lens,
+            float                 scale,
+            int                   block_size,
+            int                   max_kv_seq_len);
+
     // TurboQuant FWHT rotation. direction: 0 = forward, 1 = inverse.
     // Applies signs1 -> FWHT -> signs2 (forward) or signs2 -> FWHT -> signs1 (inverse).
     // Used for KV cache rotation in TurboQuant quantization types (TQ3_0).

server/deps/llama.cpp/ggml/src/ggml-backend-meta.cpp

@@ -730,6 +730,18 @@ static struct ggml_backend_meta_split_state ggml_backend_meta_get_split_state(co
         return {GGML_BACKEND_SPLIT_AXIS_1, {0}, 1};
     };
 
+    auto handle_paged_attn = [&](const std::vector<ggml_backend_meta_split_state> & src_ss) -> ggml_backend_meta_split_state {
+        // Paged attention currently rejects layer-split targets, so the meta
+        // backend can only see a fully mirrored op. Add sharded semantics when
+        // paged layer splitting is implemented and can be exercised end to end.
+        GGML_ASSERT(src_ss[0].axis == GGML_BACKEND_SPLIT_AXIS_MIRRORED);
+        GGML_ASSERT(src_ss[1].axis == GGML_BACKEND_SPLIT_AXIS_MIRRORED);
+        GGML_ASSERT(src_ss[2].axis == GGML_BACKEND_SPLIT_AXIS_MIRRORED);
+        GGML_ASSERT(src_ss[3].axis == GGML_BACKEND_SPLIT_AXIS_MIRRORED);
+        GGML_ASSERT(src_ss[4].axis == GGML_BACKEND_SPLIT_AXIS_MIRRORED);
+        return src_ss[0];
+    };
+
     auto handle_ssm_conv = [&](const std::vector<ggml_backend_meta_split_state> & src_ss) -> ggml_backend_meta_split_state {
         if (src_ss[0].axis == src_ss[1].axis) {
             if (src_ss[0].axis == GGML_BACKEND_SPLIT_AXIS_0) {
@@ -948,6 +960,9 @@ static struct ggml_backend_meta_split_state ggml_backend_meta_get_split_state(co
                 // multi-device split (all sources on same device as output).
                 split_state = handle_flash_attn_ext(src_ss);
             } break;
+            case GGML_OP_PAGED_ATTN: {
+                split_state = handle_paged_attn(src_ss);
+            } break;
             case GGML_OP_FLASH_ATTN_BACK: {
                 split_state = handle_generic(src_ss, /*scalar_only =*/ true);
             } break;

server/deps/llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c

@@ -2187,6 +2187,10 @@ static void ggml_compute_forward(struct ggml_compute_params * params, struct ggm
             {
                 GGML_ABORT("GGML_OP_FLASH_ATTN_SPARSE is only supported on the CUDA backend");
             }
+        case GGML_OP_PAGED_ATTN:
+            {
+                GGML_ABORT("GGML_OP_PAGED_ATTN is only supported on the CUDA backend");
+            }
         case GGML_OP_FLASH_ATTN_BACK:
             {
                 int32_t t = ggml_get_op_params_i32(tensor, 0);
@@ -2577,6 +2581,7 @@ static int ggml_get_n_tasks(struct ggml_tensor * node, int n_threads) {
         case GGML_OP_DS4_INDEXER_MASK:
         case GGML_OP_FLASH_ATTN_EXT:
         case GGML_OP_FLASH_ATTN_SPARSE:
+        case GGML_OP_PAGED_ATTN:
         case GGML_OP_FLASH_ATTN_BACK:
         case GGML_OP_SSM_CONV:
         case GGML_OP_SSM_SCAN:

server/deps/llama.cpp/ggml/src/ggml-cpu/ggml-cpu.cpp

@@ -470,6 +470,8 @@ static bool ggml_backend_cpu_device_supports_op(ggml_backend_dev_t dev, const st
             // DS4 attention carries sparse-layout and fused-RoPE semantics.
             // The generic CPU kernel does not implement that contract.
             return !ggml_flash_attn_ext_is_ds4(op);
+        case GGML_OP_PAGED_ATTN:
+            return false;
         case GGML_OP_OUT_PROD:
             return (src0->type == GGML_TYPE_F32 || (ggml_is_quantized(src0->type) && src0->ne[2] == src1->ne[2] && src0->ne[3] == src1->ne[3])) &&
                 src1->type == GGML_TYPE_F32 && op->type == GGML_TYPE_F32;

server/deps/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu

@@ -24,6 +24,7 @@
 #include "ggml-cuda/diag.cuh"
 #include "ggml-cuda/fattn.cuh"
 #include "ggml-cuda/fattn-sparse.cuh"
+#include "ggml-cuda/paged-attn.cuh"
 #include "ggml-cuda/getrows.cuh"
 #include "ggml-cuda/turbo-wht.cuh"
 #include "ggml-cuda/im2col.cuh"
@@ -3144,6 +3145,9 @@ static bool ggml_cuda_compute_forward(ggml_backend_cuda_context & ctx, struct gg
         case GGML_OP_FLASH_ATTN_SPARSE:
             ggml_cuda_flash_attn_sparse(ctx, dst);
             break;
+        case GGML_OP_PAGED_ATTN:
+            ggml_cuda_paged_attn(ctx, dst);
+            break;
         case GGML_OP_CROSS_ENTROPY_LOSS:
             ggml_cuda_cross_entropy_loss(ctx, dst);
             break;
@@ -5575,6 +5579,8 @@ static bool ggml_backend_cuda_device_supports_op(ggml_backend_dev_t dev, const g
             return ggml_cuda_flash_attn_ext_supported(dev_ctx->device, op);
         case GGML_OP_FLASH_ATTN_SPARSE:
             return true;  // Always supported on CUDA
+        case GGML_OP_PAGED_ATTN:
+            return ggml_cuda_paged_attn_supported(op);
         case GGML_OP_CROSS_ENTROPY_LOSS:
         case GGML_OP_CROSS_ENTROPY_LOSS_BACK:
         case GGML_OP_OPT_STEP_ADAMW: