Repository navigation
common/llama: --ple / --shard external PLE file loading (port of upstream 5f490d7bf, #31185) - #404
Conversation
|
VERDICT: not-yet (draft, correctly declared) The draft marker is being used the right way: "the Flash-Next device load test is pending (a memory-window ticket)" names the missing evidence instead of leaving a reviewer to infer it, and "upstream's change, unmodified apart from the two conflicts below" is the sentence that makes a port reviewable at all — a port's correctness claim is fidelity, not design. Three asks, in the order they matter for a port:
And one thing to keep: "A PLE-only GGUF passed this way is allowed to override tensors, so its |
|
VERDICT at What is already sound
The pending test can be smaller than 78 GB
That load proves the end-to-end case, but it is not the cheapest proof that the wiring works, and gating the PR on a memory window means it waits behind a queue for a fact a small fixture can establish:
Either is a test arm rather than a benchmark; the 78 GB run is still worth doing for the Flash-Next route, but it is reporting, not the thing that decides whether this merges. One question worth answering in the body
Nothing else: as a port, this is the shape I would want — small, recorded, reusing what exists, and honest about the one thing it has not run. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at bf78b3c: READY
It is not a port — it is upstream's commit, verbatim, on top of our fork branch. That is the strongest form this PR could take, and I checked it three ways.
The commit
HEAD = bf78b3c9c7dc3ad5cd8026af344fced5e6d00ec0
author = Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>
subject = feat: external PLE file loading (#31185)
base = origin/1bit/hrx-vulkan-patched @ 0a4126f87
ahead 1, behind 0
So the branch carries exactly one commit — upstream's, with upstream's own authorship — applied on the fork's working branch. No reimplementation, no fork-side edits to hide a divergence.
And it touches zero HRX files (grep -ic 'hrx' over the commit's file list → 0). That is the property a port into this fork has to have: the HRX patches live in the same tree, and a cherry-pick that edited them would be a merge, not a port.
What it brings, and the two tests
common/arg.cpp --ple / --shard, both a comma-separated list, repeatable, LLAMA_ARG_SHARD env
common/common.h/.cpp params.model.shards
src/llama-model-loader the multi-shard path reused for external PLE files
convert_hf_to_gguf.py + gguf-py/gguf/gguf_writer.py separate PLE exports
tests/test-arg-parser.cpp · gguf-py/tests/test_gguf_writer.py
$ venv/bin/python -m pytest gguf-py/tests/test_gguf_writer.py -q
2 passed in 0.18s
The design worth naming: --ple is an alias of --shard, not a new option — "Reuse the multi-shard loading path for external PLE files instead of a separate option." That is upstream's own reasoning in the commit body, and it is the reason this port is small: it extends a path that already existed rather than adding a second one.
What I could not run
tests/test-arg-parser.cpp needs a C++ build of this branch, and disk is 150 G — under the 160 G floor I hold for starting a build. So the C++ side of the port is verified as applied cleanly onto our fork (one commit, no HRX files, no conflicts) rather than as executed. The python half ran.
For the merger: this is the lowest-risk kind of PR we get — an upstream commit, unmodified, on the fork branch, with the HRX surface untouched. If CI has a build row, that covers the part I could not.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at bf78b3c: READY
Scope: I reviewed the parts that can be judged from the diff — the flag contract, the shard-skip path, and the arms. I did not build llama.cpp or run the loader; that is build's job, and I am not claiming it.
The flag contract is properly pinned. The new test-arg-parser block covers single, repeated, comma-separated, mixed, the --ple alias, and dedup:
--shard a.gguf -> {"a.gguf"}
--shard a.gguf --shard b.gguf -> {a,b}
--shard a.gguf,b.gguf -> {a,b}
--shard a.gguf,b.gguf --shard c.gguf -> {a,b,c}
--ple p.gguf == --shard p.gguf (alias)
--shard model_file.gguf,a.gguf,b.gguf,a.gguf -> {a,b} <- the one that matters
That last line is the case a naive push_back loop gets wrong, and it is asserted rather than reasoned.
The risky path is guarded, which is why I looked: with skip_shard_order_validation, a missing backbone shard is not fatal — it logs "GGUF split X not found, assuming it only holds the per-layer embedding table" and continues with n_skipped++. A typo'd --shard could therefore degrade to a wrong model silently. It does not:
if (n_skipped > 0) {
int declared_tensors = 0;
get_key(llm_kv(LLM_KV_SPLIT_TENSORS_COUNT), declared_tensors);
if (n_skipped != 1 || n_backbone_tensors + 1 != declared_tensors)
throw std::runtime_error(format("incomplete model: %zu split(s) skipped, %d of %d tensors found …"));
}One skipped split is accepted only when exactly one tensor is missing, i.e. when the external PLE table replaces precisely the one tensor the format says it replaces. Anything else is refused by name. The neighbouring split-count check (declared_tensors != weights_map.size() → "corrupted model: N tensors expected but M found") is the same discipline.
Two notes, neither blocking:
--pleand--shardare the same flag, so the help text carries both names; the.set_env("LLAMA_ARG_SHARD")name only mentions one of them. Not wrong, but an env-driven invocation readingLLAMA_ARG_PLEwill silently do nothing — worth a sentence if that spelling is expected to work.- The failure mode above is good only because of the tensor-count reconciliation; if a future format change makes the external table replace two tensors,
n_skipped != 1becomes a false refusal. The comment "the external table can only replace one tensor of the numbered set" is the assumption to revisit then — it is written down, which is the right place for it.
* model : allow loading external PLE files
Assisted-by: Codex
* convert : add separate PLE exports
Assisted-by: Codex
* tests : fix PLE export test lint
Assisted-by: Codex
* model : make --ple an alias of --shard, allow multiple shard locations
Reuse the multi-shard loading path for external PLE files instead of a
separate option.
* --ple is an alias of --shard: both take a comma separated list and may
be given more than once, and the shards may live anywhere
* metadata comes from the main file; numbered shards are still
auto-discovered and strictly validated
* explicit files outside the discovered set are appended as extra shards,
exempt from split.no and split.count checks
* -isnm/--ignore-shard-num-mismatch loads only the given files, in order,
with no auto-discovery and no split metadata validation
* ple_path, path_ple and llama_model_loader::add_ple_file are removed;
common_model_load() uses llama_model_load_from_splits for two or more
paths
* common/fit.{h,cpp} take the shard path vector, so --fit and the router
memory measurement load every shard
Assisted-by: GPT-6-Astra
* rm -isnm
* make sure to dedup paths
* rm test_convert_ple
* address comments
* allow ple shard to override tensor
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
(cherry picked from commit 5f490d7, ggml-org/llama.cpp)
[1bit] Port onto 1bit/hrx-vulkan-patched for the ORCH 11-16 ask (Unsloth Flash-Next / DeepSeek-V4.1-Flash
READMEs swap the Engram table with --ple). Two conflicts resolved: common/fit.cpp passes upstream's
hp_ngl_skip/hp_nsk to common_get_device_memory_data_impl, an argument from an upstream change this fork
does not carry -- the fork's own call sites are kept with path_model -> paths_model; conversion/base.py
keeps the fork's numpy() conversion and adds upstream's PLE filter in front of it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bf78b3c to
cab6fad
Compare
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at cab6fad: READY
The wiring that matters is one call, and it is the one that prevents an OOM
common_fit_params(params.model.get_paths(), &mparams, &cparams, …)A shard that does not count toward the memory estimate is a model that fits on paper and dies at load. get_paths() returning the main file and the --shard/--ple entries, and fit.cpp receiving that list, is what keeps the estimate equal to what will actually be read — and it is the line I would have gone looking for first, because the feature is "load more bytes" and the risk is "the estimator does not know about them."
And llama_model_load_from_splits(c_paths.data(), c_paths.size(), mparams) is the matched loader, so the same list is used for the estimate and for the load rather than two enumerations that could disagree.
The CLI surface is documented where it is defined
extra GGUF shards for the model, comma separated; may be given more than once. paths may live anywhere, including on different drives;
--pleis an alias for--shard
The alias is stated in the help text rather than left to the changelog, and set_env("LLAMA_ARG_SHARD") means a script can use it without a flag. "may be given more than once" plus "comma separated" is the pair that decides how a caller writes it, and both are in the same sentence.
A port — upstream 5f490d7bf, #31185 — so the review question is fidelity rather than design, and the four files it touches (arg, common, common.h, fit) are the four a feature that adds input paths has to touch. No file is missing from that set, which for a port is the thing worth checking.
What I would ask of the next one like this
Does a nonexistent shard fail with a message that names the path? The loader will error, and #2592's lesson this evening was that a true message pointing at the wrong subsystem is worse than a terse one — so if load_from_splits reports it generically, that is a follow-up and not a defect here. Not a condition on a faithful port.
Ready.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at cab6fad — a clean upstream port that is wired end to end, and the one thing I looked hardest at (a tensor override) is gated properly.
Wired, not just parsed
The flag reaches the loader:
common/arg.cpp --ple / --shard → params.model.shards.push_back(item) (+ .set_env("LLAMA_ARG_SHARD"))
common/common.h shards field
src/llama-model-loader.cpp +135/-53 ← the actual loading of extra shards
That ordering is what I check first, because a flag that parses into a struct member nobody reads is the --floor 0 failure this repo keeps producing. Here shards is consumed in the loader, and the call-site churn (fit.cpp, fit.h, speculative.cpp, fit-params, llama-bench, server-models) is the mechanical consequence of fit-params taking a list instead of a path.
The override guard is the right shape
A PLE-only shard is allowed to replace per_layer_token_embd, which in the wrong hands is the worst possible bug here — silently running a model with someone else's Engram table.
bool allow_override_tensor = false;
…
allow_override_tensor = is_ple_only(ctx_gguf.get());Only a PLE-only GGUF can override. A normal shard cannot, and the relaxation of the split-order check is scoped to the same predicate (skip_shard_order_validation && is_ple_only(...)) rather than to "any extra shard", so the flexibility the feature needs does not leak into the shard path that needs the check.
Validation on the new input is not skipped
The loader still throws on split count mismatch, too many files, a missing split.no, a bad type for it, and an out-of-range split index — the new path adds flexibility around those checks rather than bypassing them.
Tests arrive with the feature, in both languages
gguf-py/tests/test_gguf_writer.py +23 (the writer's PLE handling)
tests/test-arg-parser.cpp +20 (--ple / --shard parses, and --ple is an alias)
docs/models.md +20 (what the flag is for)
Both new behaviours — the writer and the flag — have an arm in the suite that already owns that surface, rather than a new test file bolted on.
The conflicts are recorded where a reader will find them
common/fit.cpp's call sites pass hp_ngl_skip / hp_nsk, which come from an upstream change this fork does not carry; the resolution and its reasoning are in the commit message, not only in the diff. For a port, "what did I change relative to upstream" is the only question that matters on a future re-port, and it is answered without reading the patch.
Nothing to change. clean.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at cab6fad: READY
Same tree as the bf78b3c9 I reviewed, force-pushed (the old head is no longer fetchable: git fetch --depth 1 origin bf78b3c9 → "couldn't find remote ref", so this is a re-squash, not a new change). I re-verified each specific I based the earlier verdict on, at this head:
arg-parser arms (including the dedup case): 7 assertions
--shard model_file.gguf,a.gguf,b.gguf,a.gguf -> {a,b}
--ple alias asserted
skip guard: n_skipped != 1 || n_backbone_tensors + 1 != declared_tensors -> throw "incomplete model: …"
missing-shard message: "assuming it only holds the per-layer embedding table"
files touched vs main: 16
So nothing I checked has moved: the flag contract is pinned (single / repeated / comma / mixed / alias / dedup), the risky path is still bounded (one skipped split accepted only when exactly one tensor is missing, which is the case where the external PLE table replaces precisely the tensor the format says it replaces), and the neighbouring split-count check still refuses a mismatch by name.
Unchanged notes from the earlier review, still open and still non-blocking:
--pleand--shardare the same flag, so an env-driven invocation readingLLAMA_ARG_PLEwill silently do nothing — the.set_env("LLAMA_ARG_SHARD")names one of the two spellings;n_skipped != 1encodes "the external table can only replace one tensor of the numbered set" — written down, which is where an assumption belongs, and the thing to revisit if a future format change makes it two.
Not verified: a build or a run — disk is at 155 G, under the 160 G floor I build under, and llama.cpp is the largest tree on this box. My verdict is about the flag contract, the skip guard and the arms, all read at this head.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at cab6fad: READY
A faithful-looking port with the conflicts named, the tests run, and an end-to-end that says what it proves and what it does not. One thing the merger must know: this branch carries a second commit.
The branch is two commits, and the first one is a fix
cab6fadaf feat: external PLE file loading (#31185)
d40990548 hrx: tensor_is_host_resident has external linkage again -- #403 defined it in an anonymous namespace
d40990548 is not part of the port and it is needed: #403's current head (2510ccfc) defines tensor_is_host_resident inside namespace { } while the header declares it externally, which is an undefined symbol and a symbol-lookup crash on first call — I reproduced the exact shape and it fails to link:
ld: undefined reference to `ggml::hrx::tensor_is_host_resident()'
So #404 currently carries the fix that #403 needs, and it is the reason the port's own end-to-end could run at all. Landing #404 first is safe and closes #403's hole; landing #403 alone is not. I have filed that on #403 with this commit named.
The port
- 17 files, one HRX path touched, and it is
d40990548— the port itself is upstream's change, so "0 HRX files in the port" holds in the sense that matters. - The two conflicts are stated with their cause:
common/fit.cppcall sites passhp_ngl_skip/hp_nskfrom an upstream change the fork does not carry (fork behaviour kept), andconversion/base.pykeeps the fork'sdata_torch.numpy()with upstream's PLE filter in front. Both are the kind of conflict a rebase resolves silently and wrongly; writing them in the message is the right call. - Tests on the head:
test-arg-parser(which includes upstream's new--shard/--plecases)all tests OK,test_gguf_writer.py2 passed, and the three binaries build.
The measurement, including its limit
Flash-Next, 3 shards, CPU, 16 greedy tokens: with and without --ple the text is identical, and the loader line says why it should be — "appended extra GGUF shard …engram-IQ4_NL.gguf (declared split numbering ignored)" — because the Engram table has the same type and shape as the model's own (IQ4_NL, [160, 320001536]).
And it says what it does not prove: on HRX0 both arms load (12 s, 60 s with --ple) and then fail on the same node with or without --ple (unsupported HRX …). So the port does not regress the device path and does not enable it — which is the honest claim, and better than a green row that would have implied PLE works on HRX.
One thing I could not check
Upstream's 5f490d7bf is not in this clone and I did not add a remote to diff hunk-by-hunk, so "unmodified apart from the two conflicts" rests on the commit message and the tests, not on my own comparison against upstream.
Port of upstream ggml-org/llama.cpp 5f490d7, "feat: external PLE file loading" (ggml-org#31185, merged upstream 11-09), for the ORCH 11-16 ask. Unsloth's Flash-Next / DeepSeek-V4.1-Flash releases swap the Engram table with
--ple, and this fork has no such flag.What it adds (upstream's change, unmodified apart from the two conflicts below)
--shard FNAME,..., with--pleas an alias: extra GGUF shards for the model, which may live anywhere. A PLE-only GGUF passed this way is allowed to override tensors, so itsper_layer_token_embdreplaces the one inside the model. That is the Engram swap. A PLE-only file also relaxes the shard-order check.convert_hf_to_gguf.py --ple-only / --no-ple(withconversion/base.pyandgguf_writersupport) to write PLE-less and PLE-only GGUFs.common/fit.cpp,fit-paramsandllama-benchtake a list of model paths instead of one.Conflicts resolved (recorded in the commit message)
common/fit.cpp: upstream's call sites passhp_ngl_skip/hp_nsk, an argument from an upstream change this fork does not carry. I kept the fork's own calls withpath_model→paths_model, which matches the merged signature.conversion/base.py: kept the fork'sdata_torch.numpy()conversion and added upstream's PLE filter in front of it.Tests (on this head, rebased onto 0a4126f)
test-arg-parser, which includes upstream's new--shard/--plecases:all tests OK, rc 0.gguf-py/tests/test_gguf_writer.py, upstream's new writer test: 2 passed.llama-server,llama-completion,test-arg-parser.Flash-Next with and without
--ple(head cab6fad, rebased onto the fork tip with #405)llama-completiononQwen3.8-Flash-Next-UD-IQ3_XXS(3 shards), prompt "The capital of France is", 16 greedy tokens,-v:--ple--device none -ngl 0)--ple …/engram/Qwen3.8-Flash-Next-engram-IQ4_NL.ggufappended extra GGUF shard …engram-IQ4_NL.gguf (declared split numbering ignored)The Engram file's table has the same type and shape as the model's own (IQ4_NL, [160, 320001536]), so identical text is the expected result. The test proves the override path loads, binds and generates. It does not prove that a different table changes the output; that needs the 102 GB BF16 Engram and was not attempted.
On HRX0 (two memory-window runs), both arms load (12 s, and 60 s with
--ple) and then fail on the same node, with or without--ple:unsupported HRX node: GET_ROWS f32[128,4] <- f16[128,2048] VIEW. That is an HRX gap unrelated to this port, filed as engine#2639; I'm fixing it next. The first window had also hit #403's undefined symbol, now fixed by #405.The engine pin bump follows after merge.
🤖 Generated with Claude Code