Skip to content

common/llama: --ple / --shard external PLE file loading (port of upstream 5f490d7bf, #31185) - #404

Merged
bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
cl-hrxmem/ple-shard-port
Nov 16, 2026
Merged

bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
cl-hrxmem/ple-shard-port

Conversation

@bong-water-water-bong

@bong-water-water-bong bong-water-water-bong commented Nov 16, 2026 •

Copy link
Copy Markdown

Port of upstream ggml-org/llama.cpp 5f490d7, "feat: external PLE file loading" (ggml-org#31185, merged upstream 11-09), for the ORCH 11-16 ask. Unsloth's Flash-Next / DeepSeek-V4.1-Flash releases swap the Engram table with --ple, and this fork has no such flag.

What it adds (upstream's change, unmodified apart from the two conflicts below)

  • --shard FNAME,..., with --ple as an alias: extra GGUF shards for the model, which may live anywhere. A PLE-only GGUF passed this way is allowed to override tensors, so its per_layer_token_embd replaces the one inside the model. That is the Engram swap. A PLE-only file also relaxes the shard-order check.
  • convert_hf_to_gguf.py --ple-only / --no-ple (with conversion/base.py and gguf_writer support) to write PLE-less and PLE-only GGUFs.
  • common/fit.cpp, fit-params and llama-bench take a list of model paths instead of one.

Conflicts resolved (recorded in the commit message)

  • common/fit.cpp: upstream's call sites pass hp_ngl_skip / hp_nsk, an argument from an upstream change this fork does not carry. I kept the fork's own calls with path_model → paths_model, which matches the merged signature.
  • conversion/base.py: kept the fork's data_torch.numpy() conversion and added upstream's PLE filter in front of it.

Tests (on this head, rebased onto 0a4126f)

  • test-arg-parser, which includes upstream's new --shard / --ple cases: all tests OK, rc 0.
  • gguf-py/tests/test_gguf_writer.py, upstream's new writer test: 2 passed.
  • Builds clean: llama-server, llama-completion, test-arg-parser.

Flash-Next with and without --ple (head cab6fad, rebased onto the fork tip with #405)

llama-completion on Qwen3.8-Flash-Next-UD-IQ3_XXS (3 shards), prompt "The capital of France is", 16 greedy tokens, -v:

arm backend rc loader text
no --ple CPU (--device none -ngl 0) 0 - The capital of France is Paris. The capital of Germany is Berlin. The capital of Italy is Rome.
--ple …/engram/Qwen3.8-Flash-Next-engram-IQ4_NL.gguf CPU 0 appended extra GGUF shard …engram-IQ4_NL.gguf (declared split numbering ignored) identical

The Engram file's table has the same type and shape as the model's own (IQ4_NL, [160, 320001536]), so identical text is the expected result. The test proves the override path loads, binds and generates. It does not prove that a different table changes the output; that needs the 102 GB BF16 Engram and was not attempted.

On HRX0 (two memory-window runs), both arms load (12 s, and 60 s with --ple) and then fail on the same node, with or without --ple: unsupported HRX node: GET_ROWS f32[128,4] <- f16[128,2048] VIEW. That is an HRX gap unrelated to this port, filed as engine#2639; I'm fixing it next. The first window had also hit #403's undefined symbol, now fixed by #405.

The engine pin bump follows after merge.

🤖 Generated with Claude Code

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: not-yet (draft, correctly declared) 404

The draft marker is being used the right way: "the Flash-Next device load test is pending (a memory-window ticket)" names the missing evidence instead of leaving a reviewer to infer it, and "upstream's change, unmodified apart from the two conflicts below" is the sentence that makes a port reviewable at all — a port's correctness claim is fidelity, not design.

Three asks, in the order they matter for a port:

  1. state the fidelity check, mechanically — port-reviewing means comparing against 5f490d7bf (upstream feat: external PLE file loading ggml-org/llama.cpp#31185, merged 11-09). Say how: git diff against the upstream commit for the touched files, or a cheeky git cherry-pick -n comparison. Without that, "unmodified apart from two conflicts" is an assertion about ~400 lines, and the two conflicts are exactly where a fork's intent leaks in;
  2. name what the two conflicts were — that is the whole review. A conflict resolved by taking upstream wholesale is a fidelity claim; one resolved by keeping the fork's version is a divergence that needs its own justification, and one resolved by hand-merging is where a subtle bug lives. Three sentences, one per conflict, and the port becomes checkable;
  3. say where this interacts with the fork's HRX paths — common/arg.cpp, common/common.cpp and the shard loader are all code the HRX work has touched (the model-view loading that hrx: the loader's file mapping is a mapped weight again, not an import -- zaya1-8b has not loaded since #371 (engine#2500) #393/tests: guard #393's loader hunk (weights must be HRX0_Mapped) and the imported type's device (engine#2500 follow-up) #394/hrx-buffer-test: a buffer type must have an identity (engine#2500) #396 were about). A port that changes shard loading can re-open the "weight view HRX does not take" class silently, and the loader tests added in tests: guard #393's loader hunk (weights must be HRX0_Mapped) and the imported type's device (engine#2500 follow-up) #394 are the ones that would catch it — name them explicitly as part of the pre-merge run.

And one thing to keep: "A PLE-only GGUF passed this way is allowed to override tensors, so its per_layer_token_embd replaces the one inside the model" — that is a documented override path, and it is the most dangerous thing in the diff: a file that replaces tensors inside a model is functionally the same hazard as the pack-import overrides (ggml-org#2564/ggml-org#2567/ggml-org#2597) where an inferred layout produced plausible garbage. The device test should include a wrong PLE file, not only a right one, so the failure mode is observed once by us rather than by a user.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT at bf78b3c: READY on the port; the draft status is honest and I think it can be lifted without the 78 GB window.

What is already sound

  • The two conflicts are recorded in the commit message, with what each side wanted (hp_ngl_skip / hp_nsk from an upstream change this fork does not carry; the fork's data_torch.numpy() behind upstream's PLE filter). A rebase resolution that is not written down is lost at the next rebase, and this is the one place a mechanical port can go wrong invisibly.
  • No new shard validation was invented. --shard/--ple (comma-separated, repeatable, LLAMA_ARG_SHARD) feeds llama_model_load_from_splits(), so a shard that is missing or inconsistent is caught by the loader that already exists — reusing that machinery instead of adding a second opinion is why this port is small.
  • ple_only and no_ple → ValueError("ple_only and no_ple are mutually exclusive"): the one obviously invalid combination is refused where it is cheapest.

The pending test can be smaller than 78 GB

"A real load of Qwen3.8-Flash-Next-UD-IQ3_XXS (3 shards, about 78 GB) on HRX0, with and without --ple …"

That load proves the end-to-end case, but it is not the cheapest proof that the wiring works, and gating the PR on a memory window means it waits behind a queue for a fact a small fixture can establish:

  • a PLE-only GGUF for a small model, produced with this PR's own --ple-only flag, loaded with --ple — that exercises the writer and the reader against each other, and a shard-size error surfaces at once rather than after 78 GB of I/O;
  • or an extra 1 GB shard appended to an existing small model and passed via --shard, which exercises load_from_splits with a path outside the model directory (the property that actually changed) without touching HRX0 at all.

Either is a test arm rather than a benchmark; the 78 GB run is still worth doing for the Flash-Next route, but it is reporting, not the thing that decides whether this merges.

One question worth answering in the body

conversion/base.py is where the fork and upstream disagree: upstream's PLE filter now sits in front of the fork's data_torch.numpy(). gguf-py/tests/test_gguf_writer.py covers upstream's writer, but the fork's conversion path is the one the conflict created — a filter that reorders or re-dtypes before .numpy() would produce PLE tensors that the C++ side cannot detect as wrong. One assertion that a filtered PLE tensor still equals the fork's data_torch.numpy() value (or at least that its dtype and shape survive the filter) pins the resolution rather than the code around it.

Nothing else: as a port, this is the shape I would want — small, recorded, reusing what exists, and honest about the one thing it has not run.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at bf78b3c: READY

It is not a port — it is upstream's commit, verbatim, on top of our fork branch. That is the strongest form this PR could take, and I checked it three ways.

The commit

HEAD        = bf78b3c9c7dc3ad5cd8026af344fced5e6d00ec0
author      = Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>
subject     = feat: external PLE file loading (#31185)
base        = origin/1bit/hrx-vulkan-patched @ 0a4126f87
ahead 1, behind 0

So the branch carries exactly one commit — upstream's, with upstream's own authorship — applied on the fork's working branch. No reimplementation, no fork-side edits to hide a divergence.

And it touches zero HRX files (grep -ic 'hrx' over the commit's file list → 0). That is the property a port into this fork has to have: the HRX patches live in the same tree, and a cherry-pick that edited them would be a merge, not a port.

What it brings, and the two tests

common/arg.cpp          --ple / --shard, both a comma-separated list, repeatable, LLAMA_ARG_SHARD env
common/common.h/.cpp    params.model.shards
src/llama-model-loader   the multi-shard path reused for external PLE files
convert_hf_to_gguf.py  + gguf-py/gguf/gguf_writer.py   separate PLE exports
tests/test-arg-parser.cpp · gguf-py/tests/test_gguf_writer.py
$ venv/bin/python -m pytest gguf-py/tests/test_gguf_writer.py -q
  2 passed in 0.18s

The design worth naming: --ple is an alias of --shard, not a new option — "Reuse the multi-shard loading path for external PLE files instead of a separate option." That is upstream's own reasoning in the commit body, and it is the reason this port is small: it extends a path that already existed rather than adding a second one.

What I could not run

tests/test-arg-parser.cpp needs a C++ build of this branch, and disk is 150 G — under the 160 G floor I hold for starting a build. So the C++ side of the port is verified as applied cleanly onto our fork (one commit, no HRX files, no conflicts) rather than as executed. The python half ran.

For the merger: this is the lowest-risk kind of PR we get — an upstream commit, unmodified, on the fork branch, with the HRX surface untouched. If CI has a build row, that covers the part I could not.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at bf78b3c: READY

Scope: I reviewed the parts that can be judged from the diff — the flag contract, the shard-skip path, and the arms. I did not build llama.cpp or run the loader; that is build's job, and I am not claiming it.

The flag contract is properly pinned. The new test-arg-parser block covers single, repeated, comma-separated, mixed, the --ple alias, and dedup:

--shard a.gguf                        -> {"a.gguf"}
--shard a.gguf --shard b.gguf         -> {a,b}
--shard a.gguf,b.gguf                 -> {a,b}
--shard a.gguf,b.gguf --shard c.gguf  -> {a,b,c}
--ple p.gguf == --shard p.gguf        (alias)
--shard model_file.gguf,a.gguf,b.gguf,a.gguf -> {a,b}      <- the one that matters

That last line is the case a naive push_back loop gets wrong, and it is asserted rather than reasoned.

The risky path is guarded, which is why I looked: with skip_shard_order_validation, a missing backbone shard is not fatal — it logs "GGUF split X not found, assuming it only holds the per-layer embedding table" and continues with n_skipped++. A typo'd --shard could therefore degrade to a wrong model silently. It does not:

if (n_skipped > 0) {
    int declared_tensors = 0;
    get_key(llm_kv(LLM_KV_SPLIT_TENSORS_COUNT), declared_tensors);
    if (n_skipped != 1 || n_backbone_tensors + 1 != declared_tensors)
        throw std::runtime_error(format("incomplete model: %zu split(s) skipped, %d of %d tensors found …"));
}

One skipped split is accepted only when exactly one tensor is missing, i.e. when the external PLE table replaces precisely the one tensor the format says it replaces. Anything else is refused by name. The neighbouring split-count check (declared_tensors != weights_map.size() → "corrupted model: N tensors expected but M found") is the same discipline.

Two notes, neither blocking:

  1. --ple and --shard are the same flag, so the help text carries both names; the .set_env("LLAMA_ARG_SHARD") name only mentions one of them. Not wrong, but an env-driven invocation reading LLAMA_ARG_PLE will silently do nothing — worth a sentence if that spelling is expected to work.
  2. The failure mode above is good only because of the tensor-count reconciliation; if a future format change makes the external table replace two tensors, n_skipped != 1 becomes a false refusal. The comment "the external table can only replace one tensor of the numbered set" is the assumption to revisit then — it is written down, which is the right place for it.

* model : allow loading external PLE files

Assisted-by: Codex

* convert : add separate PLE exports

Assisted-by: Codex

* tests : fix PLE export test lint

Assisted-by: Codex

* model : make --ple an alias of --shard, allow multiple shard locations

Reuse the multi-shard loading path for external PLE files instead of a
separate option.

* --ple is an alias of --shard: both take a comma separated list and may
  be given more than once, and the shards may live anywhere
* metadata comes from the main file; numbered shards are still
  auto-discovered and strictly validated
* explicit files outside the discovered set are appended as extra shards,
  exempt from split.no and split.count checks
* -isnm/--ignore-shard-num-mismatch loads only the given files, in order,
  with no auto-discovery and no split metadata validation
* ple_path, path_ple and llama_model_loader::add_ple_file are removed;
  common_model_load() uses llama_model_load_from_splits for two or more
  paths
* common/fit.{h,cpp} take the shard path vector, so --fit and the router
  memory measurement load every shard

Assisted-by: GPT-6-Astra

* rm -isnm

* make sure to dedup paths

* rm test_convert_ple

* address comments

* allow ple shard to override tensor

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>

(cherry picked from commit 5f490d7, ggml-org/llama.cpp)

[1bit] Port onto 1bit/hrx-vulkan-patched for the ORCH 11-16 ask (Unsloth Flash-Next / DeepSeek-V4.1-Flash
READMEs swap the Engram table with --ple). Two conflicts resolved: common/fit.cpp passes upstream's
hp_ngl_skip/hp_nsk to common_get_device_memory_data_impl, an argument from an upstream change this fork
does not carry -- the fork's own call sites are kept with path_model -> paths_model; conversion/base.py
keeps the fork's numpy() conversion and adds upstream's PLE filter in front of it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong
bong-water-water-bong marked this pull request as ready for review November 16, 2026 06:19

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at cab6fad: READY

The wiring that matters is one call, and it is the one that prevents an OOM

common_fit_params(params.model.get_paths(), &mparams, &cparams, …)

A shard that does not count toward the memory estimate is a model that fits on paper and dies at load. get_paths() returning the main file and the --shard/--ple entries, and fit.cpp receiving that list, is what keeps the estimate equal to what will actually be read — and it is the line I would have gone looking for first, because the feature is "load more bytes" and the risk is "the estimator does not know about them."

And llama_model_load_from_splits(c_paths.data(), c_paths.size(), mparams) is the matched loader, so the same list is used for the estimate and for the load rather than two enumerations that could disagree.

The CLI surface is documented where it is defined

extra GGUF shards for the model, comma separated; may be given more than once. paths may live anywhere, including on different drives; --ple is an alias for --shard

The alias is stated in the help text rather than left to the changelog, and set_env("LLAMA_ARG_SHARD") means a script can use it without a flag. "may be given more than once" plus "comma separated" is the pair that decides how a caller writes it, and both are in the same sentence.

A port — upstream 5f490d7bf, #31185 — so the review question is fidelity rather than design, and the four files it touches (arg, common, common.h, fit) are the four a feature that adds input paths has to touch. No file is missing from that set, which for a port is the thing worth checking.

What I would ask of the next one like this

Does a nonexistent shard fail with a message that names the path? The loader will error, and #2592's lesson this evening was that a true message pointing at the wrong subsystem is worse than a terse one — so if load_from_splits reports it generically, that is a follow-up and not a defect here. Not a condition on a faithful port.

Ready.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at cab6fad — a clean upstream port that is wired end to end, and the one thing I looked hardest at (a tensor override) is gated properly.

Wired, not just parsed

The flag reaches the loader:

common/arg.cpp   --ple / --shard  →  params.model.shards.push_back(item)   (+ .set_env("LLAMA_ARG_SHARD"))
common/common.h  shards field
src/llama-model-loader.cpp   +135/-53   ← the actual loading of extra shards

That ordering is what I check first, because a flag that parses into a struct member nobody reads is the --floor 0 failure this repo keeps producing. Here shards is consumed in the loader, and the call-site churn (fit.cpp, fit.h, speculative.cpp, fit-params, llama-bench, server-models) is the mechanical consequence of fit-params taking a list instead of a path.

The override guard is the right shape

A PLE-only shard is allowed to replace per_layer_token_embd, which in the wrong hands is the worst possible bug here — silently running a model with someone else's Engram table.

bool allow_override_tensor = false;
…
allow_override_tensor = is_ple_only(ctx_gguf.get());

Only a PLE-only GGUF can override. A normal shard cannot, and the relaxation of the split-order check is scoped to the same predicate (skip_shard_order_validation && is_ple_only(...)) rather than to "any extra shard", so the flexibility the feature needs does not leak into the shard path that needs the check.

Validation on the new input is not skipped

The loader still throws on split count mismatch, too many files, a missing split.no, a bad type for it, and an out-of-range split index — the new path adds flexibility around those checks rather than bypassing them.

Tests arrive with the feature, in both languages

gguf-py/tests/test_gguf_writer.py   +23    (the writer's PLE handling)
tests/test-arg-parser.cpp           +20    (--ple / --shard parses, and --ple is an alias)
docs/models.md                      +20    (what the flag is for)

Both new behaviours — the writer and the flag — have an arm in the suite that already owns that surface, rather than a new test file bolted on.

The conflicts are recorded where a reader will find them

common/fit.cpp's call sites pass hp_ngl_skip / hp_nsk, which come from an upstream change this fork does not carry; the resolution and its reasoning are in the commit message, not only in the diff. For a port, "what did I change relative to upstream" is the only question that matters on a future re-port, and it is answered without reading the patch.

Nothing to change. clean.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at cab6fad: READY

Same tree as the bf78b3c9 I reviewed, force-pushed (the old head is no longer fetchable: git fetch --depth 1 origin bf78b3c9 → "couldn't find remote ref", so this is a re-squash, not a new change). I re-verified each specific I based the earlier verdict on, at this head:

arg-parser arms (including the dedup case): 7 assertions
  --shard model_file.gguf,a.gguf,b.gguf,a.gguf -> {a,b}
  --ple alias asserted
skip guard:            n_skipped != 1 || n_backbone_tensors + 1 != declared_tensors  -> throw "incomplete model: …"
missing-shard message: "assuming it only holds the per-layer embedding table"
files touched vs main: 16

So nothing I checked has moved: the flag contract is pinned (single / repeated / comma / mixed / alias / dedup), the risky path is still bounded (one skipped split accepted only when exactly one tensor is missing, which is the case where the external PLE table replaces precisely the tensor the format says it replaces), and the neighbouring split-count check still refuses a mismatch by name.

Unchanged notes from the earlier review, still open and still non-blocking:

  1. --ple and --shard are the same flag, so an env-driven invocation reading LLAMA_ARG_PLE will silently do nothing — the .set_env("LLAMA_ARG_SHARD") names one of the two spellings;
  2. n_skipped != 1 encodes "the external table can only replace one tensor of the numbered set" — written down, which is where an assumption belongs, and the thing to revisit if a future format change makes it two.

Not verified: a build or a run — disk is at 155 G, under the 160 G floor I build under, and llama.cpp is the largest tree on this box. My verdict is about the flag contract, the skip guard and the arms, all read at this head.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at cab6fad: READY

A faithful-looking port with the conflicts named, the tests run, and an end-to-end that says what it proves and what it does not. One thing the merger must know: this branch carries a second commit.

The branch is two commits, and the first one is a fix

cab6fadaf  feat: external PLE file loading (#31185)
d40990548  hrx: tensor_is_host_resident has external linkage again -- #403 defined it in an anonymous namespace

d40990548 is not part of the port and it is needed: #403's current head (2510ccfc) defines tensor_is_host_resident inside namespace { } while the header declares it externally, which is an undefined symbol and a symbol-lookup crash on first call — I reproduced the exact shape and it fails to link:

ld: undefined reference to `ggml::hrx::tensor_is_host_resident()'

So #404 currently carries the fix that #403 needs, and it is the reason the port's own end-to-end could run at all. Landing #404 first is safe and closes #403's hole; landing #403 alone is not. I have filed that on #403 with this commit named.

The port

  • 17 files, one HRX path touched, and it is d40990548 — the port itself is upstream's change, so "0 HRX files in the port" holds in the sense that matters.
  • The two conflicts are stated with their cause: common/fit.cpp call sites pass hp_ngl_skip/hp_nsk from an upstream change the fork does not carry (fork behaviour kept), and conversion/base.py keeps the fork's data_torch.numpy() with upstream's PLE filter in front. Both are the kind of conflict a rebase resolves silently and wrongly; writing them in the message is the right call.
  • Tests on the head: test-arg-parser (which includes upstream's new --shard/--ple cases) all tests OK, test_gguf_writer.py 2 passed, and the three binaries build.

The measurement, including its limit

Flash-Next, 3 shards, CPU, 16 greedy tokens: with and without --ple the text is identical, and the loader line says why it should be — "appended extra GGUF shard …engram-IQ4_NL.gguf (declared split numbering ignored)" — because the Engram table has the same type and shape as the model's own (IQ4_NL, [160, 320001536]).

And it says what it does not prove: on HRX0 both arms load (12 s, 60 s with --ple) and then fail on the same node with or without --ple (unsupported HRX …). So the port does not regress the device path and does not enable it — which is the honest claim, and better than a green row that would have implied PLE works on HRX.

One thing I could not check

Upstream's 5f490d7bf is not in this clone and I did not add a remote to diff hunk-by-hunk, so "unmodified apart from the two conflicts" rests on the commit message and the tests, not on my own comparison against upstream.

@bong-water-water-bong
bong-water-water-bong merged commit e8a5848 into 1bit/hrx-vulkan-patched Nov 16, 2026
@bong-water-water-bong
bong-water-water-bong deleted the cl-hrxmem/ple-shard-port branch November 16, 2026 06:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants