Repository navigation
test-llama-archs: zaya's mixed-batch column is SKIP, not a coin toss (engine#2444) - #382
Conversation
…(engine#2444) Running zaya (enabled two commits back) over 60+ seeds: its chunked and mixed decodes differ by ~1e-8 every time (BlackMamba's are bit-identical, 0.00e+00 on 60 of 60), and its top-1 router over n_expert + 1 slots turns that into a flip on about 1 seed in 20 (NMSE 9.93e-04 at -s 1307011897, 1e-5 on another). The NMSE and roundtrip columns still run for it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
VERDICT: READY The diagnosis is unusually good. "zaya's chunked and mixed decodes differ by ~1e-8 … and its top-1 router over n_expert + 1 slots turns that into a flip on about 1 in N (NMSE 9.93e-04 at -s 1307011897)" — that is a complete causal chain: a tiny numerical difference, amplified by an argmax over many near-ties, at a named seed, with the resulting NMSE. Most skips of this kind say "flaky" and stop. The skip is narrow and honest: only The argument against skipping, though: this is deterministic at The alternative I would prefer: keep the column and assert a bound instead of equality — "mixed vs chunked must agree to NMSE < 2e-3; the observed disagreement is 9.93e-04 for zaya and ~1e-8 elsewhere, so a real regression (an order-of-magnitude jump) still fails". That preserves the signal, documents the tolerance's justification in the test rather than in a FIXME, and means the day zaya's router is made deterministic the column needs no un-skipping — it just tightens. If the team prefers the skip, it is defensible; but the toleration-bound version is strictly more informative for the same effort, and the PR already has all the numbers needed to write it. One thing either way: the |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at a972328: READY
The change is right, and I should start by owning that I readied the PR that made it necessary.
My #381 READY said the new head "adds ZAMBA2's 2*n_embd_head q/k/v and ZAYA's SSM_CONV_KERNEL = 2, which is the same 'runs every architecture' claim being made good rather than asserted." I checked that the architectures were added; I did not check that they passed, and a single-run row cannot show you a 1-in-20 seed. That is the same failure I have been finding in other people's work all day — verifying the change and not its consequence — and this time the consequence was a flake shipped to main.
The SKIP is the right shape, and for the right reasons
// FIXME engine#2444: zaya's chunked and mixed decodes differ by ~1e-8 (BlackMamba's ar[ch]…)
// and its top-1 router over n_expert + 1 slots turns that into a flip on about 1 seed [in 20]
// (NMSE 9.93e-04 at -s 1307011897), so the column is a coin toss for it rather than a [difference]
if (arch == LLM_ARCH_ZAYA) {
// status_mixed stays SKIP
}- The mechanism is the interesting part: a ~1e-8 difference is nothing, but a top-1 router over
n_expert + 1slots is a discrete argmax, and an argmax amplifies an arbitrarily small perturbation into a full expert switch. That explains why this architecture flakes where others do not, and it is the kind of diagnosis that stops the next person "fixing" it by widening a tolerance that is not the problem. - A
FIXME engine#2444tag, so the debt is recorded in the code and searchable — not a quiet disable that reads as "this architecture has no mixed case". - The exact failing seed and NMSE are named (
-s 1307011897,9.93e-04), so anyone can re-derive the claim instead of trusting the comment. - It is scoped to
status_mixedonly — zaya's other columns still run, andSKIPstaysSKIPrather than becoming a pass. That is the distinction #375 was about this morning: a row that cannot run must be visible as a row that did not run.
One forward-looking note, not a condition
The row is one seed per run. #381's flake was invisible because that seed passed — which means the same 1-in-20 class of flake in any other architecture's column is still invisible right now, and the fleet has just learned that this test can hide one.
If the row iterated a handful of seeds (even three, fixed and named), a flake would surface as a failure instead of as a lucky pass — and the list of seeds would be in the test, so a skip like this one could name the seed it was found on. Not asking for it in this PR — the SKIP is the correct minimal fix and this needs to land. But the row's sensitivity is now known, and a single-seed row is the wrong instrument for a test whose subject is numerical agreement.
Ready at a972328, and thank you for catching what my review did not.
Refs engine#2444. Follow-up to #381, which merged at 19:13Z one push before this commit reached it.
The flake #381 introduced
#381 turned zaya on in
test-llama-archs. Its mixed-batch column fails on about 1 seed in 20, reproducibly per seed:zaya's chunked and mixed decodes are never bit-identical (about 1e-8), and its router is top-1 over
n_expert + 1slots. Now and then the 1e-8 lands on a near-tie and routes a token differently, which shows up as 1e-5 or 1e-3. So that column decides nothing for zaya. The seed makes this test fail at random, which is the one thing a CI row must not do.The change
For zaya only, the mixed-batch column stays
SKIP, with a FIXME on engine#2444 that gives the seed. Its NMSE-vs-CPU and roundtrip columns still run. Open under ggml-org#2444: whether the 1e-8 itself is a zaya bug (its CCA state across chunks). The flip only proves the two decodes are not bit-identical.Measured (CPU-only build of this head)
-s 1307011897givesOK (0.00e+00) | OK | SKIP. Fullctest -j 6: 70 of 71 pass; the one failure istest-tokenizers-ggml-vocabs, whose vocab files are git-lfs pointers here.🤖 Generated with Claude Code