Skip to content

hrx: test-hrx-ops fa_prefill_full_cache.512 fails on the fork head (conflicting alternate value for the projection input) #2079

Closed

Description

@bong-water-water-bong

What happens

test-hrx-ops case fa_prefill_full_cache.512 in the llama.cpp fork aborts while it builds its plan. It is the only failing case of the suite (240 passed, 1 failed, 241 total).

plan status: conflicting alternate value for graph value 1 type 1 byte_count 2097152
tests/test-hrx-ops.cpp: requirement failed: plan.metadata.append(std::move(match.metadata), plan.status)

(The plan status: line is printed by a small change to the test helper in 1bit-MONSTER/llama.cpp#298; without it only the failed requirement shows.)

Measured

strixhalo, gfx1151, full amdclang Release build, one run each under gpu-hold, 2026-11-10:

  • fork head 2293f95 (kernels and dispatch untouched): fails as above.
  • 6ec7c26 + llama.cpp#298: fails the same way.

So it is not caused by #298 and was there before it.

What the message says

Type 1 is F16 and 2097152 bytes is the f16 copy of the 512-token projection input. The case matches the three projection MUL_MATs one at a time and appends each match to its plan (run_qwen_full_cache_prefill_flash_attention_scheduling_case). Two of the matches each publish an f16 copy of the same input under a different plan value, and the second append is refused.

Not determined: whether the case hands out plan value ids differently from the real scheduler (a stale test), or the second match really should have found the first copy (a dispatch fault that the scheduler path happens not to hit). hrx-backend-test's scheduler-driven rows with shared projections pass, which points at the test, but I did not prove it.

Done when

The case passes, or it is corrected with the reason, on the fork head.

Activity

  1. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Cause found and fixed in 1bit-MONSTER/llama.cpp#299 (cl-hrxnan, ready for review at 75e1448). It is a fault in the test: match_dispatch_at_index() already appends the match to the plan and the case appended it a second time, from the moved-from match (same plan value 36, empty name). The dispatch is not at fault. With the fix the whole suite is 239 passed, 0 failed.

  2. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Determination: the case is wrong, not the dispatch. The failing sequence hands two matches the same coverage snapshot, so the second re-produces an f16 copy the first already made — and the
    metadata refuses the duplicate. It is the issue's first hypothesis, with the chain in the source rather than inferred.

    Why the second match produces the copy again

    tests/test-hrx-ops.cpp at 2293f95:

    402  static ggml::hrx::ValueId next_plan_value(const ggml::hrx::Graph & graph, const ggml::hrx::CommandPlan & plan) {
    404      ... graph.values().size() + plan.transients.size() + plan.completion_counter_requests.size());
    434  static void match_dispatch_at_index(..., ggml::hrx::CommandPlan & plan, std::vector<bool> & covered_nodes, ...) {
    442      const DispatchMatchContext context = { graph, &graph.nodes()[root_index], root_index, covered_nodes, plan,
                                                 next_plan_value(graph, plan) };
    

    Both inputs the matcher uses to decide "has this already been produced?" — covered_nodes and next_plan_value — are snapshots taken at the call. The only thing that advances
    either is append_match_to_plan:

    428      for (size_t covered_node : match.covered_nodes) { ... covered_nodes[covered_node] = true; }
    425      REQUIRE(plan.metadata.append(std::move(match.metadata), plan.status));
    

    And the case builds two matches before appending either (:2294-2297):

    2294      ggml::hrx::DispatchMatch postprocess_match;
    2295      match_dispatch_at_index(..., postprocess_index, postprocess_match);
    2296      ggml::hrx::DispatchMatch flash_match;
    2297      match_dispatch_at_index(..., producer_index_for_tensor(..., flash_output), flash_match);
    

    So the flash match sees covered_nodes without anything the postprocess match covered, and the same next_plan_value — it therefore re-materialises the f16 copy of the shared projection
    input that postprocess just planned. Both copies carry graph value 1, F16, 2,097,152 bytes, and each has its own alternate_value id, so
    alternate_value_matches() (command-plan-metadata.cpp:20-23, which compares alternate_value among other fields) is false and
    append_alternate_value refuses the second (:78-88) with exactly

    conflicting alternate value for graph value 1 type 1 byte_count 2097152
    

    Why the scheduler never hits it

    ggml/src/ggml-hrx/dispatch/dispatch-scheduler.cpp matches one node at a time and appends inside the loop — plan_.metadata.append(...) at :334, with next_plan_value computed per
    iteration at :274,353. There is no window in which a match is built against a plan that does not yet contain its predecessor, so coverage is always current and the second match never
    re-creates a copy. That is also why hrx-backend-test's scheduler-driven rows with shared projections pass while this hand-sequenced case fails — the reporter's observation, now with the
    mechanism.

    The correction

    Move the append ahead of the next match, so the case follows the scheduler's ordering:

        ggml::hrx::DispatchMatch postprocess_match;
        match_dispatch_at_index(imported.graph, *registry, plan, covered_nodes, postprocess_index, postprocess_match);
        append_match_to_plan(plan, postprocess_match, covered_nodes);        // <- must precede the next match
        ggml::hrx::DispatchMatch flash_match;
        match_dispatch_at_index(..., flash_match);

    with a one-line comment saying why (covered_nodes and next_plan_value are snapshots; the scheduler appends per node). The alternative — a monotonic counter shared across matches —
    would also fix it but would make the hand-sequenced harness diverge further from the real one.

    What I did not do

    I did not build or run it: test-hrx-ops needs the full amdclang HRX stack, and this box is at 169 GB free against this lane's 175 GB floor for starting a build. So this is the
    issue's second exit criterion — "corrected with the reason" — with the confirmation run (test-hrx-ops --cases fa_prefill_full_cache.512, or the one-line move plus a full build) left to
    whoever has the build. If the append-order change does not make it pass, then the second hypothesis is live after all and the reported evidence here is the place to start.

  3. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Closing: fixed on main. The engine pin is 7a1f2b81c (#2092, merged 02:42Z), which contains the fix f76493c (llama.cpp#299): compare f76493c...7a1f2b81c is ahead 3, behind 0. With the fix test-hrx-ops ran 239 passed, 0 failed on the device (the PR's measurement); the case was a test fault, a match appended to its plan twice.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions