Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions docs/amd-drafters.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,29 @@ next step of engine#1780. What `dflash_forward_test` establishes:
- with a package directory (`-DONEBIT_DFLASH_DRAFT=<dir>` or `ONEBIT_TEST_DFLASH_DRAFT`), the real draft decodes and
one pass over a made-up context is finite and position-independent.

### DFlash 2 packages (engine#1781)

`load_draft()` also reads `DFlash2DraftModel` packages (`FastFlowLM/Qwen3.8-27B-DFlash2-NPU2` is the one on this box),
as generation 2 of the same loader. The config states the draft's interface under `dflash_config` and the rotation's base
under `rope_parameters`. Every projection, the fusion matrix included, is a grid of 4736-byte Q4_K chunks rather than
5120-byte q4_1 tiles. Each layer also carries two grouped convolutions, around the attention and around the MLP, and the
package carries a candidate selector (a projection of the hidden state and two vocabulary-sized codebooks).
`decode_draft()` fills the convolutions; the codebooks are read a row at a time with `decode_row()`, since each is
254 MB as f32.

The forward gains what generation 2 needs, each piece held to arithmetic written out in `dflash_forward_test`:

| piece | what it computes |
|---|---|
| `draft_conv()` | one half of the grouped convolution: a row's output sums, over the kernel's offsets, (base kernel + the row's own kernel for its group) times the row that many places earlier |
| `selector_pick()` | the selector's choice at one block row: among the target head's top candidates, the highest of logit + the pairwise term between the previous pick and the candidate |
| `DraftKvCache`, `cache_extend()`, `draft_forward_cached()` | the context's keys and values per layer, computed once per context row and reused by every later pass; grown in steps it gives exactly the single-call pass |
| `set_gemm_hook()` | an optional place for the draft's projections to run (the device route); unset, everything runs on the host |

Measured with these pieces against the served 27B, in a local loop: on a code prompt, 6.86 tokens per round and 7.6-7.8
tok/s against 4.2 tok/s plain decoding; output identical to plain greedy decoding. The loop is not part of this change;
the numbers are on engine#1781.

## PARD-2's repos are weights-only — borrow the target's tokenizer

`amd/PARD2-*` ship **`README.md`, `config.json`, `model.safetensors`, `warp_model.bin`** and nothing else: **no tokenizer
Expand Down
145 changes: 121 additions & 24 deletions npu/dflash_draft.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,73 @@ float need_float(const json& c, const char* key, const std::string& path) {

bool close_to(double a, double b) { return std::fabs(a - b) <= 1e-5 * std::max(1.0, std::fabs(b)); }

void read_target_layers(const json& from, const std::string& path, DraftConfig& o) {
if (!from.contains("target_layer_ids") || !from["target_layer_ids"].is_array() || from["target_layer_ids"].empty())
fail(path + ": no 'target_layer_ids'");
int prev = -1;
for (const auto& v : from["target_layer_ids"]) {
if (!v.is_number_integer() || v.get<int64_t>() < 0 || v.get<int64_t>() > 4096)
fail(path + ": 'target_layer_ids' holds something that is not a layer index");
const int id = int(v.get<int64_t>());
if (id <= prev) fail(path + ": 'target_layer_ids' is not strictly increasing at " + std::to_string(id));
o.target_layer_ids.push_back(id);
prev = id;
}
}

// DFlash2DraftModel: the geometry is at the top level as in any Qwen3 config, the draft's interface is under
// "dflash_config", the rotation's base under "rope_parameters", and the weight file has the family's fixed name.
DraftConfig read_generation2(const json& c, const std::string& path) {
DraftConfig o;
o.generation = 2;
o.hidden = need_int(c, "hidden_size", path);
o.intermediate = need_int(c, "intermediate_size", path);
o.layers = need_int(c, "num_hidden_layers", path);
o.heads = need_int(c, "num_attention_heads", path);
o.kv_heads = need_int(c, "num_key_value_heads", path);
o.head_dim = need_int(c, "head_dim", path);
o.vocab = need_int(c, "vocab_size", path);
o.sliding_window = need_int(c, "sliding_window", path);
o.rms_eps = need_float(c, "rms_norm_eps", path);
if (!c.contains("rope_parameters") || !c["rope_parameters"].is_object())
fail(path + ": no 'rope_parameters'");
o.rope_theta = need_float(c["rope_parameters"], "rope_theta", path + " rope_parameters");
o.embedding_scale = 1.0f;
if (!c.contains("dflash_config") || !c["dflash_config"].is_object()) fail(path + ": no 'dflash_config'");
const json& d = c["dflash_config"];
const std::string dp = path + " dflash_config";
o.block_size = need_int(d, "block_size", dp);
o.conv_kernel = need_int(d, "conv_kernel_size", dp);
o.conv_group = need_int(d, "conv_group_size", dp);
o.selector_rank = need_int(d, "selector_rank", dp);
o.selector_top_k = need_int(d, "selector_top_k", dp);
if (!d.contains("mask_token_id") || !d["mask_token_id"].is_number_integer() || d["mask_token_id"].get<int64_t>() < 0)
fail(dp + ": no usable 'mask_token_id'");
o.mask_token_id = d["mask_token_id"].get<int64_t>();
read_target_layers(d, dp, o);
if (c.contains("is_causal")) {
if (!c["is_causal"].is_boolean()) fail(path + ": 'is_causal' is not a boolean");
o.causal = c["is_causal"].get<bool>();
}
o.draft_weight = "dflash_weight.q4nx";
if (c.contains("draft_weight")) {
if (!c["draft_weight"].is_string() || c["draft_weight"].get<std::string>().empty()) fail(path + ": 'draft_weight' is not a file name");
o.draft_weight = c["draft_weight"].get<std::string>();
if (o.draft_weight.find('/') != std::string::npos || o.draft_weight.find("..") != std::string::npos)
fail(path + ": 'draft_weight' must be a file name inside the package, got '" + o.draft_weight + "'");
}
if (o.kv_heads > o.heads || o.heads % o.kv_heads != 0)
fail(path + ": " + std::to_string(o.heads) + " heads cannot share " + std::to_string(o.kv_heads) + " key/value heads");
if (o.block_size < 2) fail(path + ": block_size " + std::to_string(o.block_size) + " drafts nothing");
if (o.conv_kernel < 1 || o.conv_group < 1 || o.hidden % o.conv_group != 0)
fail(dp + ": a convolution of kernel " + std::to_string(o.conv_kernel) + " over groups of " +
std::to_string(o.conv_group) + " does not fit hidden_size " + std::to_string(o.hidden));
if (o.selector_rank < 1 || o.selector_top_k < 1 || o.selector_top_k > o.vocab)
fail(dp + ": selector_rank or selector_top_k is not usable");
if (o.causal) fail(path + ": a causal draft is not read by this loader");
return o;
}

} // namespace

DraftConfig read_draft_config(const std::string& package_dir) {
Expand All @@ -79,15 +146,18 @@ DraftConfig read_draft_config(const std::string& package_dir) {
}
if (!c.is_object()) fail(path + ": not a JSON object");
const std::string type = c.value("model_type", json("")).is_string() ? c.value("model_type", std::string()) : "";
if (type != "dflash") fail(path + ": model_type is '" + type + "', not 'dflash'");
// The architecture decides which tensors exist. This loader knows one; a package naming another is a different
// model (DFlash2 adds a selector and a convolution), and reading it with these rules would drop those silently.
// The architecture decides which tensors exist, so it is read first. Two are known; a package naming another is a
// different model, and reading it with these rules would drop what it adds silently.
std::string arch;
if (c.contains("architectures")) {
if (!c["architectures"].is_array() || c["architectures"].empty() || !c["architectures"][0].is_string())
fail(path + ": 'architectures' is not a list of names");
const std::string arch = c["architectures"][0].get<std::string>();
if (arch != "DFlashDraftModel") fail(path + ": architecture '" + arch + "' is not read by this loader (DFlashDraftModel is)");
arch = c["architectures"][0].get<std::string>();
}
if (arch == "DFlash2DraftModel") return read_generation2(c, path);
if (type != "dflash") fail(path + ": model_type is '" + type + "', not 'dflash'");
if (!arch.empty() && arch != "DFlashDraftModel")
fail(path + ": architecture '" + arch + "' is not read by this loader (DFlashDraftModel and DFlash2DraftModel are)");
DraftConfig o;
o.hidden = need_int(c, "hidden_size", path);
o.intermediate = need_int(c, "intermediate_size", path);
Expand All @@ -103,17 +173,7 @@ DraftConfig read_draft_config(const std::string& package_dir) {
if (!c.contains("mask_token_id") || !c["mask_token_id"].is_number_integer() || c["mask_token_id"].get<int64_t>() < 0)
fail(path + ": no usable 'mask_token_id'");
o.mask_token_id = c["mask_token_id"].get<int64_t>();
if (!c.contains("target_layer_ids") || !c["target_layer_ids"].is_array() || c["target_layer_ids"].empty())
fail(path + ": no 'target_layer_ids'");
int prev = -1;
for (const auto& v : c["target_layer_ids"]) {
if (!v.is_number_integer() || v.get<int64_t>() < 0 || v.get<int64_t>() > 4096)
fail(path + ": 'target_layer_ids' holds something that is not a layer index");
const int id = int(v.get<int64_t>());
if (id <= prev) fail(path + ": 'target_layer_ids' is not strictly increasing at " + std::to_string(id));
o.target_layer_ids.push_back(id);
prev = id;
}
read_target_layers(c, path, o);
if (!c.contains("draft_weight") || !c["draft_weight"].is_string() || c["draft_weight"].get<std::string>().empty())
fail(path + ": no 'draft_weight'");
o.draft_weight = c["draft_weight"].get<std::string>();
Expand Down Expand Up @@ -172,13 +232,20 @@ DraftModel load_draft(const std::string& package_dir) {
v.cols = cols;
return v;
};
// A generation-2 package stores its projections as Q4_K chunks (4736 B per 32 x 256), not q4_1 tiles (5120 B).
const int chunk = c.generation == 2 ? Q4NX_CHUNK_Q4K : Q4NX_TILE_BYTES;
const auto projection = [&](const std::string& name, int n, int k) {
if (n % 32 != 0 || k % 256 != 0)
fail(path + ": '" + name + "' would be " + std::to_string(n) + " x " + std::to_string(k) +
", which is not a whole 32 x 256 tile grid");
return take(name, "I8", q4nx_tensor_bytes(n, k), n, k);
TensorView v = take(name, "I8", size_t(n / 32) * size_t(k / 256) * size_t(chunk), n, k);
v.chunk = chunk;
return v;
};
const auto gain = [&](const std::string& name, int width) { return take(name, "BF16", size_t(width) * 2, 1, width); };
const auto plain = [&](const std::string& name, int rows, int cols) {
return take(name, "BF16", size_t(rows) * size_t(cols) * 2, rows, cols);
};

// The container repeats the interface scalars; a package whose two halves disagree is not one model.
const auto scalar = [&](const std::string& name, const char* dtype, size_t count) {
Expand All @@ -189,7 +256,7 @@ DraftModel load_draft(const std::string& package_dir) {
const auto agree = [&](bool same, const std::string& name, const std::string& got, const std::string& want) {
if (!same) fail(path + ": '" + name + "' is " + got + " and config.json says " + want);
};
{
if (c.generation == 1) {
const uint8_t* p = scalar("dflash.block_size", "I32", 1);
agree(i32_at(p, 0) == c.block_size, "dflash.block_size", std::to_string(i32_at(p, 0)), std::to_string(c.block_size));
p = scalar("dflash.sliding_window", "I32", 1);
Expand All @@ -210,8 +277,15 @@ DraftModel load_draft(const std::string& package_dir) {
// and the declared shape must be that blocking -- a plain [hidden, taps*hidden] label would mean another element order.
if (c.hidden % 32 != 0) fail(path + ": hidden_size " + std::to_string(c.hidden) + " does not fill 32-row blocks");
const int fused = c.taps() * c.hidden;
m.fc = take("model.fc.weight", "BF16", size_t(c.hidden) * size_t(fused) * 2, c.hidden, fused);
{
if (c.generation == 2) {
m.fc = projection("model.fc.weight", c.hidden, fused);
const int groups = c.hidden / c.conv_group;
m.selector_hproj = plain("model.candidate_selector.hidden_projection.weight", c.selector_rank, c.hidden);
m.selector_pred = plain("model.candidate_selector.predecessor_codebook", c.vocab, c.selector_rank);
m.selector_succ = plain("model.candidate_selector.successor_codebook", c.vocab, c.selector_rank);
(void)groups;
} else {
m.fc = take("model.fc.weight", "BF16", size_t(c.hidden) * size_t(fused) * 2, c.hidden, fused);
const std::vector<int64_t> want = {c.hidden / 32, fused, 32};
if (by_name["model.fc.weight"]->shape != want)
fail(path + ": 'model.fc.weight' is not declared [hidden/32, taps*hidden, 32], so its element order is not the one this loader undoes");
Expand All @@ -232,11 +306,18 @@ DraftModel load_draft(const std::string& package_dir) {
L.post_norm = gain(p + "post_attention_layernorm.weight", c.hidden);
L.q_norm = gain(p + "self_attn.q_norm.weight", c.head_dim);
L.k_norm = gain(p + "self_attn.k_norm.weight", c.head_dim);
if (c.generation == 2) {
const int kernels = 2 * c.conv_kernel, groups = c.hidden / c.conv_group;
L.attn_conv_base = plain(p + "attention_conv.base_kernel", kernels, c.hidden);
L.attn_conv_proj = plain(p + "attention_conv.kernel_projection.weight", kernels * groups, c.hidden);
L.mlp_conv_base = plain(p + "mlp_conv.base_kernel", kernels, c.hidden);
L.mlp_conv_proj = plain(p + "mlp_conv.kernel_projection.weight", kernels * groups, c.hidden);
}
m.layers.push_back(L);
}
for (const auto& t : table.tensors)
if (used.count(t.name) == 0)
fail(path + ": tensor '" + t.name + "' is not part of a DFlashDraftModel (" + std::to_string(table.tensors.size() - used.size()) +
fail(path + ": tensor '" + t.name + "' is not part of this draft's architecture (" + std::to_string(table.tensors.size() - used.size()) +
" unexplained): this is another architecture");
return m;
}
Expand All @@ -246,9 +327,11 @@ std::vector<float> decode_projection(const TensorView& t) {
std::vector<float> out(size_t(t.rows) * size_t(t.cols));
// Not the size-dispatching q4nx_dequant_tensor(): a projection here is a tile grid by contract, and a grid whose
// byte count happened to equal rows*cols*2 must not be read as plain 2-byte floats.
if (t.bytes != q4nx_tensor_bytes(t.rows, t.cols) ||
q4nx_dequant_tensor_chunked(t.data, t.bytes, t.rows, t.cols, 5120, out.data()) != 0)
fail("'" + t.name + "' does not decode as a " + std::to_string(t.rows) + " x " + std::to_string(t.cols) + " q4_1 tile grid");
const int chunk = t.chunk > 0 ? t.chunk : Q4NX_TILE_BYTES;
if (t.rows % 32 != 0 || t.cols % 256 != 0 || t.bytes != size_t(t.rows / 32) * size_t(t.cols / 256) * size_t(chunk) ||
q4nx_dequant_tensor_chunked(t.data, t.bytes, t.rows, t.cols, chunk, out.data()) != 0)
fail("'" + t.name + "' does not decode as a " + std::to_string(t.rows) + " x " + std::to_string(t.cols) + " grid of " +
std::to_string(chunk) + "-byte tiles");
return out;
}

Expand All @@ -259,6 +342,20 @@ std::vector<float> decode_gain(const TensorView& t) {
return out;
}

std::vector<float> decode_plain(const TensorView& t) {
if (!t.ok() || t.bytes != size_t(t.rows) * size_t(t.cols) * 2) fail("decode_plain of '" + t.name + "': not 2 bytes per element");
std::vector<float> out(size_t(t.rows) * size_t(t.cols));
for (size_t i = 0; i < out.size(); ++i) out[i] = bf16(t.data + i * 2);
return out;
}

void decode_row(const TensorView& t, int row, float* out) {
if (!t.ok() || t.bytes != size_t(t.rows) * size_t(t.cols) * 2) fail("decode_row of '" + t.name + "': not 2 bytes per element");
if (row < 0 || row >= t.rows || out == nullptr) fail("decode_row of '" + t.name + "': row " + std::to_string(row) + " is outside the tensor");
const uint8_t* p = t.data + size_t(row) * size_t(t.cols) * 2;
for (int i = 0; i < t.cols; ++i) out[i] = bf16(p + size_t(i) * 2);
}

std::vector<float> decode_fusion(const TensorView& t) {
if (!t.ok() || t.rows % 32 != 0 || t.bytes != size_t(t.rows) * size_t(t.cols) * 2)
fail("decode_fusion of '" + t.name + "': not hidden x fused at 2 bytes per element");
Expand Down
23 changes: 22 additions & 1 deletion npu/dflash_draft.h
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,13 @@ struct DraftConfig {
float rope_theta = 0.0f;
float embedding_scale = 0.0f;
bool causal = false;
// Which draft this is. 1: DFlashDraftModel (model_type "dflash"). 2: DFlash2DraftModel, which adds a grouped
// convolution around each layer's attention and MLP and a candidate selector, states its interface under
// "dflash_config", and stores every projection (the fusion matrix included) as Q4_K chunks.
int generation = 1;
int conv_kernel = 0, conv_group = 0; // generation 2
int selector_rank = 0, selector_top_k = 0; // generation 2
int vocab = 0; // generation 2: rows of the selector's codebooks
std::string draft_weight; // the container's file name, relative to the package
std::string base_model; // the target this draft was made for, as the package names it (may be empty)

Expand All @@ -74,17 +81,25 @@ struct TensorView {
size_t bytes = 0;
int rows = 0; // N (out); 1 for a vector
int cols = 0; // K (in), or the vector's length
int chunk = 0; // a projection's bytes per 32 x 256 tile: 0 or 5120 = q4_1, 4736 = Q4_K
bool ok() const { return data != nullptr && bytes != 0; }
};

struct DraftLayer {
TensorView q, k, v, o, gate, up, down; // q4_1 tile grids, N x K
TensorView input_norm, post_norm, q_norm, k_norm; // 2-byte gains
// Generation 2, 2 bytes per element: base kernels [2][kernel][hidden], kernel projections
// [2 * kernel * hidden / group][hidden] row-major.
TensorView attn_conv_base, attn_conv_proj, mlp_conv_base, mlp_conv_proj;
};

struct DraftModel {
DraftConfig cfg;
TensorView fc; // hidden x (taps * hidden): fuses the target layers' hidden states
TensorView fc; // hidden x (taps * hidden): fuses the target layers' hidden states (generation 1: 2-byte
// elements in 32-row blocks, decode_fusion(); generation 2: a tile grid, decode_projection())
// Generation 2: the candidate selector. hidden_projection [rank][hidden], the two codebooks [vocab][rank], all at
// 2 bytes per element. The codebooks are 254 MB each as f32, so a caller reads rows out of the view (decode_row()).
TensorView selector_hproj, selector_pred, selector_succ;
TensorView hidden_norm; // applied to the fused features
TensorView norm; // the final norm
std::vector<DraftLayer> layers;
Expand All @@ -105,6 +120,12 @@ std::vector<float> decode_projection(const TensorView& t);
// A norm's gains as f32.
std::vector<float> decode_gain(const TensorView& t);

// A plain 2-byte-per-element tensor as f32, row-major [rows][cols].
std::vector<float> decode_plain(const TensorView& t);

// One row of a plain 2-byte-per-element tensor into out[cols]. Throws if the row is outside the tensor.
void decode_row(const TensorView& t, int row, float* out);

// The fusion matrix as f32, row-major [hidden][taps * hidden]. The container stores it as [hidden/32][taps*hidden][32]
// -- 32-row blocks, column by column, the row inside the block last -- and that is the order this undoes:
// out[(a*32 + r) * cols + k] = stored[(a * cols + k) * 32 + r].
Expand Down
Loading
Loading