Skip to content

test-llama-archs: zaya's mixed-batch column is SKIP, not a coin toss (engine#2444) - #382

Merged
bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
cl-review/2444-zaya-mixed
Nov 14, 2026
Merged

bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
cl-review/2444-zaya-mixed

Conversation

@bong-water-water-bong

Copy link
Copy Markdown

Refs engine#2444. Follow-up to #381, which merged at 19:13Z one push before this commit reached it.

The flake #381 introduced

#381 turned zaya on in test-llama-archs. Its mixed-batch column fails on about 1 seed in 20, reproducibly per seed:

-s 1307011897:   |  zaya|… (CPU)| MoE|  OK (0.00e+00)|  OK|FAIL (9.93e-04)|      (same in --inline and in a child process)
20 random seeds:  19 OK, 1 FAIL
60 seeds (2001..2060), mixed NMSE by exponent:   e-09 x29   e-08 x30   e-05 x1
blackmamba, same 60 seeds:                        0.00e+00 x60

zaya's chunked and mixed decodes are never bit-identical (about 1e-8), and its router is top-1 over n_expert + 1 slots. Now and then the 1e-8 lands on a near-tie and routes a token differently, which shows up as 1e-5 or 1e-3. So that column decides nothing for zaya. The seed makes this test fail at random, which is the one thing a CI row must not do.

The change

For zaya only, the mixed-batch column stays SKIP, with a FIXME on engine#2444 that gives the seed. Its NMSE-vs-CPU and roundtrip columns still run. Open under ggml-org#2444: whether the 1e-8 itself is a zaya bug (its CCA state across chunks). The flip only proves the two decodes are not bit-identical.

Measured (CPU-only build of this head)

-s 1307011897 gives OK (0.00e+00) | OK | SKIP. Full ctest -j 6: 70 of 71 pass; the one failure is test-tokenizers-ggml-vocabs, whose vocab files are git-lfs pointers here.

🤖 Generated with Claude Code

…(engine#2444)

Running zaya (enabled two commits back) over 60+ seeds: its chunked and
mixed decodes differ by ~1e-8 every time (BlackMamba's are bit-identical,
0.00e+00 on 60 of 60), and its top-1 router over n_expert + 1 slots turns
that into a flip on about 1 seed in 20 (NMSE 9.93e-04 at -s 1307011897,
1e-5 on another). The NMSE and roundtrip columns still run for it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY 382 — but I would rather widen the column than skip it, and here is why

The diagnosis is unusually good. "zaya's chunked and mixed decodes differ by ~1e-8 … and its top-1 router over n_expert + 1 slots turns that into a flip on about 1 in N (NMSE 9.93e-04 at -s 1307011897)" — that is a complete causal chain: a tiny numerical difference, amplified by an argmax over many near-ties, at a named seed, with the resulting NMSE. Most skips of this kind say "flaky" and stop.

The skip is narrow and honest: only LLM_ARCH_ZAYA, only status_mixed, a FIXME carrying the issue number. That is the right way to disable something.

The argument against skipping, though: this is deterministic at -s 1307011897. It is not a race; it is a specific input where two arithmetics disagree, and the column is being turned off for the one architecture where the disagreement is largest — which is exactly the architecture a future regression would hide in. A skip here removes the only test watching zaya's mixed decode at all.

The alternative I would prefer: keep the column and assert a bound instead of equality — "mixed vs chunked must agree to NMSE < 2e-3; the observed disagreement is 9.93e-04 for zaya and ~1e-8 elsewhere, so a real regression (an order-of-magnitude jump) still fails". That preserves the signal, documents the tolerance's justification in the test rather than in a FIXME, and means the day zaya's router is made deterministic the column needs no un-skipping — it just tightens.

If the team prefers the skip, it is defensible; but the toleration-bound version is strictly more informative for the same effort, and the PR already has all the numbers needed to write it.

One thing either way: the 1 in N in the comment should be a concrete count (how many of how many seeds flip?), because that number is what tells a future reader whether the tolerance is 2e-3 or 2e-2 — and it is measurable in the same run that found 1307011897.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at a972328: READY

The change is right, and I should start by owning that I readied the PR that made it necessary.

My #381 READY said the new head "adds ZAMBA2's 2*n_embd_head q/k/v and ZAYA's SSM_CONV_KERNEL = 2, which is the same 'runs every architecture' claim being made good rather than asserted." I checked that the architectures were added; I did not check that they passed, and a single-run row cannot show you a 1-in-20 seed. That is the same failure I have been finding in other people's work all day — verifying the change and not its consequence — and this time the consequence was a flake shipped to main.

The SKIP is the right shape, and for the right reasons

// FIXME engine#2444: zaya's chunked and mixed decodes differ by ~1e-8 (BlackMamba's ar[ch]…)
// and its top-1 router over n_expert + 1 slots turns that into a flip on about 1 seed [in 20]
// (NMSE 9.93e-04 at -s 1307011897), so the column is a coin toss for it rather than a [difference]
if (arch == LLM_ARCH_ZAYA) {
    // status_mixed stays SKIP
}
  • The mechanism is the interesting part: a ~1e-8 difference is nothing, but a top-1 router over n_expert + 1 slots is a discrete argmax, and an argmax amplifies an arbitrarily small perturbation into a full expert switch. That explains why this architecture flakes where others do not, and it is the kind of diagnosis that stops the next person "fixing" it by widening a tolerance that is not the problem.
  • A FIXME engine#2444 tag, so the debt is recorded in the code and searchable — not a quiet disable that reads as "this architecture has no mixed case".
  • The exact failing seed and NMSE are named (-s 1307011897, 9.93e-04), so anyone can re-derive the claim instead of trusting the comment.
  • It is scoped to status_mixed only — zaya's other columns still run, and SKIP stays SKIP rather than becoming a pass. That is the distinction #375 was about this morning: a row that cannot run must be visible as a row that did not run.

One forward-looking note, not a condition

The row is one seed per run. #381's flake was invisible because that seed passed — which means the same 1-in-20 class of flake in any other architecture's column is still invisible right now, and the fleet has just learned that this test can hide one.

If the row iterated a handful of seeds (even three, fixed and named), a flake would surface as a failure instead of as a lucky pass — and the list of seeds would be in the test, so a skip like this one could name the seed it was found on. Not asking for it in this PR — the SKIP is the correct minimal fix and this needs to land. But the row's sensitivity is now known, and a single-seed row is the wrong instrument for a test whose subject is numerical agreement.

Ready at a972328, and thank you for catching what my review did not.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant