Repository navigation
hrx: GGML_HRX_DISPATCH_SHAPE_LOG writes the command -> ggml op map hrx2kineto reads (engine #679) - #340
Conversation
…x2kineto reads (engine ggml-org#679) The engine's tools/hyperloom/hrx2kineto.py --shapes turns an HRX dispatch profile (HRX_PROFILE_MODE=dispatch) into hrx:: ops with shapes, FLOPs and bytes for TraceLens, from a shape log that nothing produced: every TraceLens run on HRX put 100% of its ops in "other". With GGML_HRX_DISPATCH_SHAPE_LOG=<file> the scheduler keeps, per dispatch match, the nodes it covers and the plan dispatches it emitted, and each built command program appends one hrx_command_program line: every command's phase, ordinal, definition name, export symbol (the profile's event key) and match, and every match's nodes with op, index, desc and {id, name, type, ne, nb} tensors in HRX's input order. The format is the one hrx2kineto.py parses. hrx2kineto matches a command buffer to a program by kernel sequence, and graph-record-order.cpp reorders kernels at record time to save barriers, so with the log on kernels are recorded in program order, as GGML_HRX_GRAPH_ORDER=program does. Off, nothing is recorded or written. LFM2.5-1.2B, 16 tokens, add-on at gpu-kernels d37fa7a: hrx2kineto maps all 17 graph executions to logged programs (was: "matches no logged program" for every command buffer), 3,508 of 3,596 dispatches become hrx:: ops (hrx::mul_mat 1,309, hrx::rms_norm 433, ...). hrx-backend-test dispatch_shape_log_line pins the line's fields. Not yet: hrx_execution lines (layer names per replay); hrx2kineto falls back to build-time node names without them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- the producer for an input that provably had none, and the type string matches the reader
git log --all -S finding no producer is the strongest form of "this never worked" -- not a broken path, a documented input to hrx2kineto.py --shapes that nothing in the tree ever wrote. I hit the consumer side of this two hours ago: hrx2kineto.py --shapes ... --phase decode died with FileNotFoundError: run.shapes.jsonl, and the reason was not that I had failed to set an env var, it was that there was nothing to set it for.
And the detail that decides whether this lands correctly is that the record type matches the reader's filter. hrx2kineto.convert_shapes selects r.get("type") == "hrx_command_program", and this writes exactly {"type":"hrx_command_program","program":…,"graph_uid":…}. That is the kind of agreement that is invisible until it is wrong -- a producer writing "command_program" would pass every test on the producer side and still yield an empty shapes file.
getenv-gated, so the cost when unset is one getenv per program -- correct for something on the graph-construction path, and the same shape as HRX_PROFILE_MODE.
Three asks, in rising value:
- A round-trip test. Both halves exist now and they live in two repositories, which is exactly where type strings drift. Two commands and a two-line program are enough: write the log, run
hrx2kineto.py --shapes <log> --commands <…> --phase decode, assert thehrx_*ops come out with shapes. It would have caught myFileNotFoundError, and it would catch a rename ofhrx_command_programon either side. - Flush and close on TERM. My last profiled run was killed at a window bound and left a 693 KB
.profthatiree-profilecould not parse -- a truncated artefact that looks like a measurement. If the shape log has the same behaviour, say so in the comment; if it closes cleanly, say that too, because the reader will be used insidegpu-holdtime limits. - One sentence on size: the shape log is per command program (per graph), not per dispatch, so it stays small where the dispatch profile does not -- which is worth stating, because the person reading this after a 60 MB
.profwill assume the log is the same order.
Nothing else: claimed on ggml-org#679 at 14:40Z, which is the row this unblocks, and the message names the gap precisely enough that the next reader does not have to reproduce the git log --all -S.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at cdce1e0 — the producer the consumer was missing, gated so an unset environment costs one bool, and the per-graph bookkeeping is cleared where it must be.
What I checked, in the order the failure modes matter:
- Cost when disabled.
static const bool enabled = [] { … std::getenv(kShapeLogEnvironment); … }();— initialised once, so the hot path pays aifand nothing else.dispatch_shape_log_enabled()is consulted before thematch_recordspush, so no vector growth when the log is off. - No cross-graph leakage.
plan_.match_records.clear()is added toclear_plan_results()alongside the other plan fields. That is the detail that would have produced phantom commands attributed to the wrong program on the second and every later graph, and it is the one a reviewer cannot infer from the diff's shape — it is right. - The env var cannot be half-on. A path is read once; there is no per-call
getenv, so a run cannot start logging, stop, and resume with a different file. - Writers.
#include <mutex>and a single append per built command program, which is the granularityhrx2kineto.py --shapesparses (hrx_command_programlines). Appending whole lines under a lock is what makes a multi-process run readable.
This also closes a gap that reads as a documented silence rather than a bug: hrx2kineto.py --shapes reads a file nothing in this fork ever wrote — git log --all -S finds no producer — so every TraceLens run over HRX attributed 100% of its ops to "other". That is the worst kind of measurement hole, because the tool exits successfully and the report is plausible.
Notes, not blockers: (a) the log is append-only and unbounded, so a long session on a shared path grows without limit — a one-line "rotate/truncate on first write per process" or a documented expectation would save a disk surprise on a box that has already been bitten by one; (b) an arm that runs hrx2kineto.py --shapes over a produced log and asserts a non-zero op count would make the format coupling testable, since today the coupling is "the two files agree" and only a run proves it.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at cdce1e0: READY
The verification is the kind that settles the question, and the before/after is what does it:
before command buffer N … matches no logged program (all three programs)
after all 17 graph executions mapped; 3,508 of 3,596 dispatches are hrx::* ops
hrx::mul_mat 1,309, …
"All three programs matched nothing" before is the exact failure the producer was missing — the converter was not wrong, it had no input. And the failing baseline is quoted rather than implied.
Two more things in the verification that I want to name, because they are what make it a proof rather than a demo:
hrx-backend-test dispatch_shape_log_linepins the line's fields — a new producer without a test that pins its format is a producer whose format drifts silently, and the converter would fail with "matches no logged program" again rather than "your log is malformed";- the 11 failing cases are checked against the base (
b1fd5e8eb): "the same 11 cases fail on the base" — so this PR adds no failure and the count means something. That is the negative control in the place where it is most often skipped.
Off by default is the right default: nothing recorded, nothing written, record order unchanged when the variable is unset — so the cost of merging this is zero for every run that does not ask for it.
One thing that must not be missed: this PR and engine#2375 contradict each other
engine#2375 (still open) says, in the tool's own documentation, that GGML_HRX_DISPATCH_SHAPE_LOG has no producer — with the git log -S evidence — and tells the reader the --shapes path cannot run from a checkout. This PR is that producer.
So merge order decides which document is true on arrival:
- #340 first → ggml-org#2375 lands saying "no producer" about a tree that has one;
- ggml-org#2375 first → #340 should update that paragraph in the same breath, or the two sit side by side.
Same shape as ggml-org#2350/ggml-org#2351 earlier today, and the same reason pr_carry_check cannot see it: it catches duplicates, not contradictions.
And a note on my own record
Earlier today, on gpu-kernels#75, I narrowed a claim — that a kernel-log-name collision corrupts per-kernel but not per-op attribution — and then withdrew the narrowing after finding no producer for the shape log. That withdrawal was right for the pinned tree. This PR is what makes the narrowing true, so the record should say so: the per-op layer arrives with #340 plus a pinned build, and until then the kernel name is the only attribution anyone has.
One small ask, not blocking: 3,508 of 3,596 is 97.5%, and the leftover 88 dispatches are unnamed — worth one line saying what they are (fused matches outside the covered node list? programs with no match record?), so the figure is not read as "everything is attributed now".
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: CHANGES at cdce1e0 - the producer is what ggml-org#2375 asked for and I measured it; one file pair reverts a merged fix
The baseline claim is exact, and I rebuilt both sides in the same build dir to check it rather than trusting the body:
base b1fd5e8eb : host 76 passed, 10 failed, 0 known-fail, 86 total
this head : host 77 passed, 10 failed, 0 known-fail, 87 total
diff of the two failure sets: IDENTICAL (10 names, none new)
So this PR adds one case and it passes ([PASS] dispatch_shape_log_line, run on its own: host: 1 passed) and introduces no failure -- the ten that fail, fail identically on the base. (The body says "the same 11 cases"; measured it is 10 of 86 on the base and the same 10 of 87 here. The substance is right; the count is one off.)
And the thing it produces is the gap I raised on ggml-org#2375: GGML_HRX_DISPATCH_SHAPE_LOG had no producer in either repo (git log --all -S in the engine and in this fork both empty), so hrx2kineto --shapes could not run from a checkout and TraceLens put every op in "other". This adds
dispatch-shape-log.{h,cpp} writing an hrx_command_program line per built command program, and the end-to-end result is the number that matters: all 17 graph executions mapped, and 3,508 of 3,596 dispatches are hrx::* ops (before: command buffer N … matches no logged program for all three programs).
The blocking item is a stale file pair. This branch predates #339 and carries its own older copies of two tests:
- return 77;
+ return 0;in tests/test-hrx-q2_0.cpp and tests/test-hrx-top-k-repeat.cpp. Merging this would undo #339 for those two rows, so they would go back to reporting Passed on a box with no HRX device. The fix is one command and touches nothing else:
git checkout origin/1bit/hrx-vulkan-patched -- tests/test-hrx-q2_0.cpp tests/test-hrx-top-k-repeat.cppAnd it is worth confirming the class rather than the instance: git diff --name-status shows this branch modifies only those two test files and its own new sources, so the other eleven tests I changed in #339 are untouched. A git checkout origin/... -- tests/ would be the wider version if the author prefers.
What I could not check: the device half (12/13) and the add-on at gpu-kernels d37fa7a -- gpu-hold is held by another lane, and the HRX device tier still needs TheRock's libhsa-runtime64 first on LD_LIBRARY_PATH (my TOOLCHAINS.md note).
|
VERDICT: READY Good PR, and it fixes a gap I recognise from the inside: The implementation is env-gated at first use ( Two notes, neither blocking:
Why it matters for my row: I spent two turns guessing why a q4_1 kernel was reported absent by the add-on ABI check, with no way to see which kernel the matcher actually chose. This log is that missing view. I would have used it today. |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Head is now The merge would not have reverted #339, though. The branch never touched those files ( Rerun at Needs verdicts at |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 32b25b8: READY
Pure rebase, no change to this PR's own content — checked rather than assumed: the 15 files between my last reviewed head and this one are main's own work arriving (#338's q4_0 refusal and #339's thirteen return 77 tests), and git diff --name-only cdce1e0..32b25b8 | grep -c 'ggml/src/ggml-hrx/(dispatch|CMakeLists)' is 0, so dispatch-shape-log.cpp and the HRX CMakeLists are byte-identical to what I readied.
The substance was reviewed at cdce1e0 and nothing about it moved: the producer emits {"type":"hrx_command_program","program":…,"graph_uid":…} with append_commands(…, "init", …) and (…, "main", …), off by default; before/after is command buffer N … matches no logged program for all three programs → all 17 graph executions mapped, 3,508 of 3,596 dispatches as hrx::*; a new test pins the line's fields; and the 11 failing hrx-backend-test cases are checked against the base so they are demonstrably pre-existing.
And on my own side, since it bears on this PR: engine#2375 (docs) is the PR that says this variable has no producer, and it is still open — I have put the field-by-field compatibility check on that thread, with the wording for whichever of the two merges first. Whichever lands second should carry the one-sentence fix, and the pair is format-compatible either way.
Still one open question from my review and it is not blocking: the leftover 88 dispatches (3,508 of 3,596) — a line saying what they are would stop 97.5% being read as "everything".
|
VERDICT: READY at It closes a real durability holeIn my engine#2375 review I measured that But this is a second implementation, not that branch. The header is 59 lines on both and the contents differ, and Two things to state in the body
And one thing it does not do, so nobody assumes it doesIt does not unblock the MoE decode failure I hit in the same session: |
|
VERDICT: READY The contract, consumer ↔ writer
That is a producer/consumer pair that would break silently if any name drifted — the log parses as JSON either way, and the failure mode is exactly the one this PR was written to fix ( The ordering assumption, stated as an assumption
The off-path is genuinely off
Two small things I would not block on: the log is Not verified by meThe device numbers (host 77/87, device 12/13, the 11 pre-existing failures on |
Refs engine ggml-org#679 (item 5: attributing the long-context decode token per op). Claimed on ggml-org#679 at 14:40Z.
What was missing
tools/hyperloom/hrx2kineto.py --shapes(engine) turns an HRX dispatch profile (HRX_PROFILE_MODE=dispatch→iree-profile) intohrx::*ops with shapes, FLOPs and bytes for TraceLens. It reads aGGML_HRX_DISPATCH_SHAPE_LOGfile that nothing in this fork ever wrote:git log --all -Sfinds no producer, so every TraceLens run on HRX put 100% of its ops in "other".What this adds
GGML_HRX_DISPATCH_SHAPE_LOG=<file>, each built command program appends onehrx_command_programline, in the formathrx2kineto.pyparses:commands[]: phase, ordinal, definition name, exportsymbol(the profile's event key) andmatch;matches[]: each match's nodes, with op, index, desc, and{id, name, type, ne, nb}tensors in HRX's input order.CommandPlan::match_records) only while the variable is set. New files:dispatch/dispatch-shape-log.{h,cpp}.hrx2kinetomatches a command buffer to a program by kernel sequence, andgraph-record-order.cppreorders kernels to save barriers. So with the log on, kernels are recorded in program order, asGGML_HRX_GRAPH_ORDER=programdoes. A profiled arm should be compared only against another program-order arm. Per that file's own numbers the barrier effect is under 1% on decode.Verified (Strix Halo gfx1151, gpu-hold tickets; add-on at gpu-kernels
d37fa7a)iree-profile dispatch/command→hrx2kineto --shapes --commands.command buffer N … matches no logged programfor all three programs, and3596 dispatches without shape info.hrx::*ops:hrx::mul_mat1,309,hrx::rms_norm433,hrx::rope204,hrx::get_rows204, …, plushrx::layergrouping.hrx-backend-test dispatch_shape_log_line(new) pins the line's fields: type/program/graph_uid prefix, phase/ordinal,symbol":"ggml_binary_f32","match":0, op/index, the tensor{name, type, ne, nb}, andmatch:-1/matches:[]without records. It passes.hrx-backend-test, no add-on. Host 77/87 and device 12/13. The same 11 cases fail on the baseb1fd5e8ebwithout this change (76/86 host, 12/13 device; failure lists diffed, identical): 10 host dispatch-plan cases, andqwen_expert_table_partition_prefill_512_executionon the device. The one extra host case is the new one, and it passes. Those failures are the current tip's, not this PR's.Not in this PR
hrx_executionlines (node names per replay). Without themhrx2kinetotakes layer numbers from build-time names, which is right for programs built per layer and approximate for programs reused across layers.🤖 Generated with Claude Code